Nov 03 22:22:25.917081 ip-10-0-128-222 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:22:26.375065 ip-10-0-128-222 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:26.375065 ip-10-0-128-222 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:22:26.375065 ip-10-0-128-222 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:26.375065 ip-10-0-128-222 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:26.379993 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.379901 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.379993 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.379991 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.379993 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.379995 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.379993 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.379998 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380001 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380004 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380006 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380010 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380013 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380016 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380019 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380022 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380026 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380030 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380033 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380037 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380040 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380042 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380045 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380047 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380050 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380053 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.380112 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380055 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380058 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380060 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380069 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380072 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380075 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380077 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380081 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380083 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380086 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380089 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380092 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380094 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380097 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380099 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380103 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380106 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380108 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380111 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380114 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.380559 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380117 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380119 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380122 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380125 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380128 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380131 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380134 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380157 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380161 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380164 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380167 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380170 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380172 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380175 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380178 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380180 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380183 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380186 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380189 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380193 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.381047 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380196 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380199 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380201 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380204 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380206 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380209 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380212 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380214 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380217 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380220 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380223 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380225 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380229 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380238 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380243 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380246 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380249 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380251 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380254 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.381548 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380258 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380262 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380265 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380268 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380271 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380274 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380277 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380279 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380282 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380284 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380287 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380291 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380294 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380296 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380299 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380301 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380304 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380306 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380309 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.382021 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380311 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380314 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380317 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380320 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380322 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.380325 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381128 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381135 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381138 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381141 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381144 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381147 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381164 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381169 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381172 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381175 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381178 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381181 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381183 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381186 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.382505 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381188 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381191 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381195 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381197 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381201 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381204 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381207 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381210 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381213 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381215 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381218 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381221 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381223 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381227 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381229 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381232 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381234 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381237 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381239 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381242 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.383004 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381245 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381247 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381250 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381253 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381255 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381258 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381261 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381264 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381266 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381269 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381272 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381274 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381277 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381279 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381282 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381285 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381288 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381293 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381296 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381298 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.383479 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381301 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381304 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381306 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381309 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381312 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381314 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381317 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381319 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381322 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381324 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381327 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381329 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381332 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381334 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381337 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381340 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381342 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381345 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381347 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.384009 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381350 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381353 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381355 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381358 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381360 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381365 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381369 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381373 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381376 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381379 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381382 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381386 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381390 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381393 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381396 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381399 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381402 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381404 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381407 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.384509 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381410 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381413 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381416 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381418 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381421 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381423 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381426 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381428 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381431 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381433 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381436 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381439 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381441 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381444 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381967 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381974 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381978 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381982 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381985 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.385303 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381988 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381991 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381994 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381996 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.381999 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382004 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382006 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382009 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382012 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382014 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382017 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382019 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382022 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382025 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382028 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382030 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382033 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382036 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382039 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.385845 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382041 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382044 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382046 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382049 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382052 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382054 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382057 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382059 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382062 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382064 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382067 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382070 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382072 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382075 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382078 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382081 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382083 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382086 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382088 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382091 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.386297 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382094 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382097 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382100 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382102 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382105 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382107 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382110 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382113 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382115 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382118 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382121 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382123 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382126 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382128 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382131 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382133 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382139 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382141 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382144 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.386820 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382146 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382149 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382152 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382154 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382157 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382159 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382161 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382166 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382170 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382173 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382176 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382179 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382182 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382186 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382189 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382199 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382211 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382214 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382216 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.387253 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382219 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382221 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382224 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382227 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382232 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382235 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382237 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382240 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382242 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382245 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382249 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382251 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382254 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382257 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382259 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382262 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382265 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382267 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382270 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382272 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.387707 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382275 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382278 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382280 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.382283 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.384333 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.384344 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386465 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386488 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386635 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386646 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386662 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386669 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386674 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386679 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386685 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386690 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386695 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386700 2581 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386704 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386709 2581 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386713 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386730 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386737 2581 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386742 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:22:26.388183 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386747 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386751 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386756 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386762 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386767 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386772 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386776 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386781 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386791 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386796 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386801 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386805 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386812 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386817 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386823 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386828 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386833 2581 flags.go:64] FLAG: --enable-server="true" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386838 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386846 2581 flags.go:64] FLAG: --event-burst="100" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386851 2581 flags.go:64] FLAG: --event-qps="50" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386856 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386861 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386865 2581 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386877 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:22:26.388855 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386880 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386883 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386886 2581 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386890 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386893 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386896 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386899 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386902 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386904 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386908 2581 flags.go:64] FLAG: --feature-gates="" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386912 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386915 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386919 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386922 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386926 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386929 2581 flags.go:64] FLAG: --help="false" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386932 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386935 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386938 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386941 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386945 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386950 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386953 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:22:26.389484 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386957 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386960 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386963 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386966 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386969 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386972 2581 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386975 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386977 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386980 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386983 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386986 2581 flags.go:64] FLAG: --lock-file="" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386989 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386992 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.386995 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387001 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387004 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387007 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387011 2581 flags.go:64] FLAG: --logging-format="text" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387014 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387017 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387020 2581 flags.go:64] FLAG: --manifest-url="" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387023 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387029 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387032 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387036 2581 flags.go:64] FLAG: --max-pods="110" Nov 03 22:22:26.390033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387039 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387042 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387045 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387048 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387051 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387054 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387057 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387066 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387069 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387073 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387076 2581 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387079 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387082 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387085 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387088 2581 flags.go:64] FLAG: --port="10250" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387091 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387094 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0012edbe060cd8f9c" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387097 2581 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387100 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387103 2581 flags.go:64] FLAG: --register-node="true" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387106 2581 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387110 2581 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387112 2581 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387115 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387118 2581 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387122 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:22:26.390624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387127 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387130 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387133 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387136 2581 flags.go:64] FLAG: --runonce="false" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387139 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387142 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387145 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387148 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387151 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387154 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387157 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387160 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387163 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387166 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387169 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387172 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387175 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387179 2581 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387182 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387188 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387190 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387193 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387198 2581 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387201 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387204 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:22:26.391276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387207 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387210 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387213 2581 flags.go:64] FLAG: --v="2" Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387218 2581 flags.go:64] FLAG: --version="false" Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387222 2581 flags.go:64] FLAG: --vmodule="" Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387226 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387229 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387374 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387379 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387383 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387387 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387389 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387392 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387395 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387397 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387400 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387402 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387405 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387408 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387410 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387413 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.391859 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387415 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387418 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387420 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387423 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387426 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387428 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387431 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387433 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387436 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387438 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387441 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387443 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387445 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387448 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387451 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387453 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387456 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387458 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387461 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.392367 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387464 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387467 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387470 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387473 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387475 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387478 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387481 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387483 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387486 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387488 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387491 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387494 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387496 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387499 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387501 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387504 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387507 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387509 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387512 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387515 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.392825 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387517 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387519 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387522 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387524 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387527 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387529 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387531 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387534 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387537 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387539 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387546 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387548 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387551 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387554 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387557 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387559 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387562 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387565 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387567 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387570 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.393347 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387573 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387575 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387578 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387580 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387583 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387585 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387588 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387591 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387593 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387596 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387599 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387601 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387604 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387606 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387608 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387611 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387613 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387616 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387618 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.393856 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387621 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387624 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387628 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387633 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387636 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387640 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387643 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387646 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387650 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387652 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387655 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387657 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387661 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387665 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.387693 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:26.394322 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387748 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387753 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387756 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387759 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387762 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387764 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387767 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387769 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387772 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387775 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387777 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387780 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387783 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387785 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387788 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387791 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387793 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387796 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387798 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387801 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.394693 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387803 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387807 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387810 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387813 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387815 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387818 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387822 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387825 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387828 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387831 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387834 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387836 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387839 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387841 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387844 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387846 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387849 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387852 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387854 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387857 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.395185 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387859 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387863 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387865 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387868 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387871 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387874 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387877 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387879 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387882 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387885 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387887 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387890 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387893 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387897 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387899 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387902 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387904 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387907 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387910 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387913 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.395662 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387916 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387919 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387921 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387924 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387927 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387929 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387932 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387934 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387937 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387939 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387942 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387944 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387947 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387949 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387953 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387957 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387960 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387963 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387966 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.396142 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387969 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387971 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387974 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387977 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387979 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387982 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387987 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387991 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387994 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387997 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.387999 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388003 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388006 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388009 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388012 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388014 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388017 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388019 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388022 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.396601 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388024 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388027 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388031 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388034 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388036 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388039 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388041 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.388044 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.395476 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.395491 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.395509 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.395514 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395579 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395585 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395589 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395592 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395595 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395598 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395601 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395604 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.397177 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395607 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395609 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395612 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395615 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395617 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395620 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395624 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395627 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395630 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395632 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395635 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395638 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395640 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395643 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395646 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395649 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395652 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395654 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395657 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395660 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.397642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395663 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395666 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395670 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395673 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395676 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395680 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395684 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395687 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395690 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395693 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395695 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395698 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395701 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395704 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395706 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395709 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395711 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395714 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395731 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.398121 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395735 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395738 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395740 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395743 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395746 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395748 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395751 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395754 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395757 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395759 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395762 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395765 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395768 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395771 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395775 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395777 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395780 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395783 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395785 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395788 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.398570 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395791 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395793 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395796 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395798 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395801 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395804 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395806 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395809 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395811 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395814 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395817 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395820 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395824 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395828 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395830 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395833 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395835 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395838 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395841 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.399074 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395843 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395846 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395849 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395851 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395854 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395857 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395860 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395865 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395868 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395871 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395874 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395877 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395879 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395882 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395884 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395887 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395890 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395892 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.399519 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.395895 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.395923 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396073 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396077 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396081 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396084 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396088 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396093 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396096 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396099 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396102 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396105 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396107 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396110 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396113 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.399977 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396115 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396118 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396121 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396123 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396126 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396128 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396131 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396133 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396136 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396138 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396141 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396144 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396146 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396149 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396152 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396154 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396157 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396160 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396162 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396165 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.400326 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396167 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396170 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396172 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396175 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396177 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396180 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396183 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396185 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396188 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396191 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396193 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396196 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396198 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396201 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396204 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396206 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396209 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396212 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396214 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396217 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.400858 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396219 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396222 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396224 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396227 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396229 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396237 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396241 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396245 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396247 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396250 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396253 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396255 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396258 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396261 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396263 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396266 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396268 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396271 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396273 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.401352 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396277 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396280 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396283 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396286 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396288 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396291 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396293 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396296 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396298 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396301 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396304 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396306 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396309 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396311 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396314 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396317 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396321 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396324 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396327 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.401950 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396330 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396333 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396336 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396338 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396341 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396343 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396346 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396349 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396351 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396354 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396357 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396359 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396362 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396364 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396367 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.396390 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:26.402405 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396422 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396427 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396430 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396433 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396435 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396438 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396441 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396444 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396446 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396449 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396451 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396454 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396456 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396459 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396462 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396464 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396467 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396469 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396472 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:26.402791 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396474 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396477 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396479 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396483 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396487 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396490 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396492 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396495 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396498 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396501 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396503 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396506 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396509 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396512 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396515 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396517 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396520 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396522 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396525 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:26.403286 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396527 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396530 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396533 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396535 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396538 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396540 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396543 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396546 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396550 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396553 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396556 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396558 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396564 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396566 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396569 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396572 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396575 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396577 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396580 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:26.403733 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396583 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396586 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396588 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396591 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396593 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396596 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396598 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396604 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396607 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396610 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396612 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396615 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396618 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396620 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396623 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396625 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396628 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396630 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396633 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396636 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:26.404446 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396638 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396641 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396643 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396646 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396648 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396652 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396655 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396658 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396661 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396663 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396666 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396668 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396671 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396674 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396676 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396679 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396681 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396684 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396687 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396689 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:26.405073 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396692 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396695 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396697 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396700 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396704 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396706 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396709 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396711 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:26.396714 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.396782 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.400516 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.401587 2581 server.go:1014] "Starting client certificate rotation" Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.401696 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.401767 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.403941 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:26.405594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.405479 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:26.417208 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.417187 2581 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:22:26.420740 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.420709 2581 log.go:25] "Validated CRI v1 image API" Nov 03 22:22:26.421517 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.421490 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:22:26.425851 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.425831 2581 fs.go:135] Filesystem UUIDs: map[03cd8e91-a896-4fdd-b375-c063837d526c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9171a9d5-de88-4fd4-b7eb-49e00613354e:/dev/nvme0n1p4] Nov 03 22:22:26.425904 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.425851 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:22:26.431557 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.431256 2581 manager.go:217] Machine: {Timestamp:2026-11-03 22:22:26.429556058 +0000 UTC m=+0.384575165 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098481 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c14ce8157ce113a9d2443a493ce00 SystemUUID:ec2c14ce-8157-ce11-3a9d-2443a493ce00 BootID:f6090455-3819-455a-bb6a-a637e53451d5 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f6:56:12:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f6:56:12:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:7d:69:d4:da:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:22:26.432100 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.432084 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:22:26.432207 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.432192 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:22:26.435646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.435621 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:22:26.435810 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.435647 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-222.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:22:26.435853 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.435814 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:22:26.435853 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.435823 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:22:26.435853 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.435834 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:26.435853 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.435845 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:26.436799 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.436789 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:22:26.440884 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.440871 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:22:26.441007 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.440998 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:22:26.441812 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.441803 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:22:26.441850 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.441819 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:22:26.441850 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.441829 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:22:26.441850 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.441838 2581 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:22:26.441850 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.441845 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:22:26.442187 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.442164 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:26.444914 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.444892 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:22:26.451507 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.451490 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:26.452391 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452377 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:22:26.452445 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452409 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:22:26.452566 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452556 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:22:26.452597 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452572 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:22:26.452597 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452579 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:22:26.452597 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452584 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:22:26.452597 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452590 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:22:26.452597 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.452596 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:22:26.453340 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.453330 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:22:26.453340 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.453340 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:22:26.453394 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.453347 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:22:26.453394 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.453353 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:22:26.453394 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.453362 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:22:26.453394 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.453371 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:22:26.454141 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.454129 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:22:26.454187 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.454144 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:22:26.458085 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.458072 2581 server.go:1263] "Started kubelet" Nov 03 22:22:26.458175 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.458149 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:22:26.458285 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.458234 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:22:26.458339 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.458310 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:22:26.460059 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.460039 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:22:26.460523 ip-10-0-128-222 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:22:26.463308 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.462952 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:22:26.465679 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.465653 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:22:26.467696 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.467679 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:22:26.468915 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.468897 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:22:26.469035 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.469017 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" Nov 03 22:22:26.470064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.469748 2581 factory.go:55] Registering systemd factory Nov 03 22:22:26.470064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.469765 2581 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:22:26.470064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.469806 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:22:26.470064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.469824 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:22:26.470064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.469946 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:22:26.470064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.469953 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:22:26.470064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.469973 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:22:26.470310 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.470068 2581 factory.go:153] Registering CRI-O factory Nov 03 22:22:26.470310 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.470082 2581 factory.go:223] Registration of the crio container factory successfully Nov 03 22:22:26.470310 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.470147 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:22:26.470310 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.470166 2581 factory.go:103] Registering Raw factory Nov 03 22:22:26.470310 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.470179 2581 manager.go:1199] Started watching for new ooms in manager Nov 03 22:22:26.471348 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.471333 2581 manager.go:319] Starting recovery of all containers Nov 03 22:22:26.481681 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.481663 2581 manager.go:324] Recovery completed Nov 03 22:22:26.486136 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.486120 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:26.489375 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.489341 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:26.489463 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.489394 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:26.489463 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.489411 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:26.490021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.490005 2581 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:22:26.490021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.490020 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:22:26.490134 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.490040 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:22:26.490134 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.490044 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:26.490839 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.490820 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:26.490897 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.490847 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:26.490897 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.490860 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:26.492246 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.492229 2581 policy_none.go:50] "Start" Nov 03 22:22:26.492283 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.492250 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:22:26.492283 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.492260 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:22:26.494359 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.494348 2581 policy_none.go:44] "Start" Nov 03 22:22:26.541306 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.541274 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:22:26.544517 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.544501 2581 manager.go:341] "Starting Device Plugin manager" Nov 03 22:22:26.544616 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.544542 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:22:26.544616 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.544555 2581 server.go:87] "Starting device plugin registration server" Nov 03 22:22:26.544864 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.544842 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:22:26.544961 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.544861 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:22:26.545060 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.545044 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:22:26.545165 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.545152 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:22:26.545165 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.545165 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:22:26.547154 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.546973 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:22:26.547222 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.547196 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" Nov 03 22:22:26.603593 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.603555 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:22:26.605131 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.604797 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:22:26.605131 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.604818 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:22:26.605131 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.604841 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:22:26.605131 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.604873 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 22:22:26.606431 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.606179 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-222.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:22:26.606431 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.606187 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-222.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:22:26.607898 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.606184 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-222.us-west-2.compute.internal.18e4a9294d6cfcad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-222.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-222.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:22:26.458041517 +0000 UTC m=+0.413060624,LastTimestamp:2026-11-03 22:22:26.458041517 +0000 UTC m=+0.413060624,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-222.us-west-2.compute.internal,}" Nov 03 22:22:26.645782 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.645702 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:26.646768 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.646745 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:26.646879 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.646782 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:26.646879 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.646793 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:26.646879 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.646820 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.659750 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.659716 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:22:26.663277 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.663257 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:22:26.663363 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.663331 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7ckml" Nov 03 22:22:26.706233 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.706191 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal"] Nov 03 22:22:26.706346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.706281 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:26.707285 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.707267 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:26.707404 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.707298 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:26.707404 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.707313 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:26.708829 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.708813 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:26.708968 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.708949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.709018 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.708983 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:26.709565 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.709546 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:26.709654 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.709578 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:26.709654 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.709595 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:26.709654 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.709549 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:26.709654 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.709648 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:26.709856 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.709660 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:26.711187 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.711167 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.711187 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.711190 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:26.711909 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.711893 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:26.711986 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.711918 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:26.711986 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.711929 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:26.739044 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.739016 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.742635 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:26.742609 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.771129 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.771098 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.771286 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.771128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.771286 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.771165 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d79e0f53b5b82fd1a135f4636735856a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"d79e0f53b5b82fd1a135f4636735856a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.774917 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.774891 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:26.777168 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.777152 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:26.872258 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.872218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.872258 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.872258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.872469 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.872279 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d79e0f53b5b82fd1a135f4636735856a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"d79e0f53b5b82fd1a135f4636735856a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.872469 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.872351 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d79e0f53b5b82fd1a135f4636735856a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"d79e0f53b5b82fd1a135f4636735856a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.872469 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.872353 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.872469 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.872351 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.882035 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.882012 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:26.882561 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.882538 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:26.969670 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:26.969576 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:28.491682 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.491652 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:28.494604 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.494588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:28.543132 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.543101 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:28.544132 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.544118 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:22:28.645596 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.645564 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:28.849515 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.849419 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:22:28.850066 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.849600 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:22:28.850066 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.849676 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:22:28.850066 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:28.849676 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a16baf3e91ce6443fadccbde99d65a93-20d6be8647d4ee72.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-222.us-west-2.compute.internal.18e4a9294f4b8320\": read tcp 10.0.128.222:38266->18.222.12.232:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-222.us-west-2.compute.internal.18e4a9294f4b8320 default 5472 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-222.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-222.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-222.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:22:26 +0000 UTC,LastTimestamp:2026-11-03 22:22:26.646787916 +0000 UTC m=+0.601807024,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-222.us-west-2.compute.internal,}" Nov 03 22:22:28.890521 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.890490 2581 apiserver.go:52] "Watching apiserver" Nov 03 22:22:28.916270 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:28.916236 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:22:29.116573 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.116487 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:17:28 +0000 UTC" deadline="2028-08-13 11:28:51.269662522 +0000 UTC" Nov 03 22:22:29.116573 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.116526 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15565h6m22.153140371s" Nov 03 22:22:29.136248 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.136217 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:22:29.136546 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.136524 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zlkqx","openshift-cluster-node-tuning-operator/tuned-g2gcl","openshift-image-registry/node-ca-95xrs","kube-system/konnectivity-agent-lrdk5","kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5","openshift-multus/multus-6rv4x","openshift-network-diagnostics/network-check-target-hdq68","openshift-dns/node-resolver-bhsqf","openshift-network-operator/iptables-alerter-c54qw","openshift-ovn-kubernetes/ovnkube-node-fq4b6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-x22cf","openshift-multus/network-metrics-daemon-xfjg6"] Nov 03 22:22:29.138762 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.138746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.138859 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.138841 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:29.140166 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.140151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.141585 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.141568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.143083 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.143069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.144708 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.144693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.146319 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.146307 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.147845 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.147829 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:29.147922 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.147900 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:29.149403 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.149388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.151085 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.151062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.152895 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.152878 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.155043 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.155028 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.156958 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.156942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:29.157024 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.156995 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:29.188656 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.188632 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.188656 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.188648 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.188828 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.188684 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hvj2k\"" Nov 03 22:22:29.189709 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.189692 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kfchj\"" Nov 03 22:22:29.189709 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.189693 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:22:29.189856 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.189698 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.189856 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.189740 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.190960 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.190945 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:22:29.191162 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.191149 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:22:29.191222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.191190 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ztkgs\"" Nov 03 22:22:29.192725 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.192711 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.192817 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.192744 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hp7n8\"" Nov 03 22:22:29.192817 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.192771 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.192909 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.192833 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:22:29.193967 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.193954 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.194031 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.194020 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:22:29.218960 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.218942 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:22:29.226295 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:29.226389 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-netns\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.226389 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-systemd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.226389 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-netd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.226389 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-system-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-netns\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9tx\" (UniqueName: \"kubernetes.io/projected/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-kube-api-access-xz9tx\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-etc-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226556 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908bcf40-a8a2-4655-8561-473df410dfb0-ovn-node-metrics-cert\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6sz\" (UniqueName: \"kubernetes.io/projected/1ac28a61-63eb-4ac5-a145-73045bf8111d-kube-api-access-7w6sz\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.226615 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-run\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrqs\" (UniqueName: \"kubernetes.io/projected/41972913-8aa1-4d44-97b0-033ba10133f5-kube-api-access-4jrqs\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-kubelet\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-ovn\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-script-lib\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-socket-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-bin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-env-overrides\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226863 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-conf-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-kubernetes\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-log-socket\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r484\" (UniqueName: \"kubernetes.io/projected/b6adef5f-a8df-4321-a2c3-ea90ca555c52-kube-api-access-2r484\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.226992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-var-lib-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227021 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9443ca41-9780-4ecc-94c4-a1d99f138a59-host\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-multus\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227054 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-hostroot\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-multus-certs\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-etc-kubernetes\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv8z7\" (UniqueName: \"kubernetes.io/projected/48fecb37-0646-4895-9958-ac184f9356bd-kube-api-access-fv8z7\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41972913-8aa1-4d44-97b0-033ba10133f5-hosts-file\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cnibin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-kubelet-config\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7q4p\" (UniqueName: \"kubernetes.io/projected/908bcf40-a8a2-4655-8561-473df410dfb0-kube-api-access-n7q4p\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227323 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227348 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227365 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-modprobe-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-slash\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.227646 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-node-log\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227512 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-systemd\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227540 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-sys\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpg9\" (UniqueName: \"kubernetes.io/projected/6b9b949c-4919-4d59-ae4d-e0083278cfed-kube-api-access-kcpg9\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227591 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9443ca41-9780-4ecc-94c4-a1d99f138a59-serviceca\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-socket-dir-parent\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227647 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-kubelet\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de70e8cd-d2c4-45b4-bcd4-9a726497f781-konnectivity-ca\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227691 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cni-binary-copy\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-daemon-config\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-dbus\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-conf\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227811 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-bin\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dp7\" (UniqueName: \"kubernetes.io/projected/9443ca41-9780-4ecc-94c4-a1d99f138a59-kube-api-access-j2dp7\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de70e8cd-d2c4-45b4-bcd4-9a726497f781-agent-certs\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.228396 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-lib-modules\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.227988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-var-lib-kubelet\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-tuned\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-tmp\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-iptables-alerter-script\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-host-slash\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-k8s-cni-cncf-io\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysconfig\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-host\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-systemd-units\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-config\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228237 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-registration-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hzt\" (UniqueName: \"kubernetes.io/projected/01c3b879-44a4-435d-ae16-acc5c57992b0-kube-api-access-66hzt\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.229257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228333 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-cnibin\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.230069 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-device-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.230069 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228407 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-sys-fs\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.230069 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41972913-8aa1-4d44-97b0-033ba10133f5-tmp-dir\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.230069 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.228455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-system-cni-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.286678 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.286655 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:22:29.286786 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.286715 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sltvf" Nov 03 22:22:29.289156 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.289141 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:22:29.289685 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.289663 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:22:29.289685 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.289662 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:22:29.289685 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.289682 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.289864 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.289686 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:22:29.289971 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.289955 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.289971 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.289965 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:22:29.290056 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290044 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.290103 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290059 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nh9f8\"" Nov 03 22:22:29.290103 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290077 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.290103 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290076 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kccnl\"" Nov 03 22:22:29.290103 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290100 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.290279 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290174 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.290279 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290184 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.290279 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290214 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6sdx\"" Nov 03 22:22:29.290279 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290238 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:22:29.290279 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290263 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:22:29.290279 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290264 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mdpv5\"" Nov 03 22:22:29.290461 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290291 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ljhlr\"" Nov 03 22:22:29.290461 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.290326 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:22:29.329192 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329163 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-system-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.329346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-netns\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.329346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329237 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-system-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.329346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9tx\" (UniqueName: \"kubernetes.io/projected/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-kube-api-access-xz9tx\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.329346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-etc-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-netns\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.329346 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329327 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908bcf40-a8a2-4655-8561-473df410dfb0-ovn-node-metrics-cert\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329361 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329369 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-etc-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329382 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6sz\" (UniqueName: \"kubernetes.io/projected/1ac28a61-63eb-4ac5-a145-73045bf8111d-kube-api-access-7w6sz\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-run\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329402 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrqs\" (UniqueName: \"kubernetes.io/projected/41972913-8aa1-4d44-97b0-033ba10133f5-kube-api-access-4jrqs\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-kubelet\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-ovn\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-script-lib\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-socket-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-bin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329572 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-run\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329579 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-ovn\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-env-overrides\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.329728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329617 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-kubelet\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-conf-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329654 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-bin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-kubernetes\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329690 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329736 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-kubernetes\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-log-socket\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329760 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-socket-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329777 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r484\" (UniqueName: \"kubernetes.io/projected/b6adef5f-a8df-4321-a2c3-ea90ca555c52-kube-api-access-2r484\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329691 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-conf-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329822 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-var-lib-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9443ca41-9780-4ecc-94c4-a1d99f138a59-host\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329869 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-multus\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329898 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-multus\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329901 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-hostroot\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329782 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-log-socket\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.330536 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-multus-certs\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329949 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9443ca41-9780-4ecc-94c4-a1d99f138a59-host\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-etc-kubernetes\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329987 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-hostroot\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.329991 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-etc-kubernetes\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330029 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-multus-certs\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv8z7\" (UniqueName: \"kubernetes.io/projected/48fecb37-0646-4895-9958-ac184f9356bd-kube-api-access-fv8z7\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330133 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-env-overrides\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330152 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41972913-8aa1-4d44-97b0-033ba10133f5-hosts-file\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330200 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-var-lib-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cnibin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cnibin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330253 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41972913-8aa1-4d44-97b0-033ba10133f5-hosts-file\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-kubelet-config\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330302 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7q4p\" (UniqueName: \"kubernetes.io/projected/908bcf40-a8a2-4655-8561-473df410dfb0-kube-api-access-n7q4p\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.331380 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330349 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-kubelet-config\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330365 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-modprobe-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330410 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-slash\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330445 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-slash\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-node-log\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330494 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-node-log\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-systemd\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330527 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-sys\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpg9\" (UniqueName: \"kubernetes.io/projected/6b9b949c-4919-4d59-ae4d-e0083278cfed-kube-api-access-kcpg9\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330563 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-systemd\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9443ca41-9780-4ecc-94c4-a1d99f138a59-serviceca\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330561 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-modprobe-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-socket-dir-parent\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330630 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-sys\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.332222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330646 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-kubelet\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de70e8cd-d2c4-45b4-bcd4-9a726497f781-konnectivity-ca\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330690 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-script-lib\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cni-binary-copy\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-daemon-config\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330741 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-kubelet\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-dbus\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-conf\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330821 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330858 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-dbus\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-bin\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330903 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-conf\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.330981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dp7\" (UniqueName: \"kubernetes.io/projected/9443ca41-9780-4ecc-94c4-a1d99f138a59-kube-api-access-j2dp7\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331013 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de70e8cd-d2c4-45b4-bcd4-9a726497f781-agent-certs\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-lib-modules\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331064 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9443ca41-9780-4ecc-94c4-a1d99f138a59-serviceca\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.333049 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-var-lib-kubelet\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331133 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-tuned\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-tmp\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331176 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-socket-dir-parent\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-iptables-alerter-script\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331239 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-lib-modules\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-host-slash\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331317 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-daemon-config\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-k8s-cni-cncf-io\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysconfig\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-host\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331295 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-var-lib-kubelet\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331331 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cni-binary-copy\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331399 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de70e8cd-d2c4-45b4-bcd4-9a726497f781-konnectivity-ca\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-systemd-units\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331456 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-k8s-cni-cncf-io\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.333579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-config\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331493 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331514 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-bin\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-registration-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331540 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysconfig\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hzt\" (UniqueName: \"kubernetes.io/projected/01c3b879-44a4-435d-ae16-acc5c57992b0-kube-api-access-66hzt\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331555 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-host\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-host-slash\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-registration-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331649 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331667 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-cnibin\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-device-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-sys-fs\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41972913-8aa1-4d44-97b0-033ba10133f5-tmp-dir\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331751 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331781 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-iptables-alerter-script\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.334095 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-system-cni-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.331879 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331910 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-netns\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331914 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-cnibin\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331856 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-sys-fs\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331882 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-systemd-units\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.331961 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret podName:b2e63ee2-e60b-45fd-90e4-e60785022ff7 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:29.831926634 +0000 UTC m=+2.338862637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret") pod "global-pull-secret-syncer-zlkqx" (UID: "b2e63ee2-e60b-45fd-90e4-e60785022ff7") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332001 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-device-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.331880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-netns\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-systemd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-netd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332141 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-systemd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332176 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-netd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332217 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-system-cni-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.332289 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.332342 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs podName:48fecb37-0646-4895-9958-ac184f9356bd nodeName:}" failed. No retries permitted until 2026-11-03 22:22:29.832328836 +0000 UTC m=+2.339264834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs") pod "network-metrics-daemon-xfjg6" (UID: "48fecb37-0646-4895-9958-ac184f9356bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:29.334559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332751 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41972913-8aa1-4d44-97b0-033ba10133f5-tmp-dir\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.335057 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.332785 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-config\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.335057 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.333011 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.335057 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.333575 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908bcf40-a8a2-4655-8561-473df410dfb0-ovn-node-metrics-cert\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.335057 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.334298 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-tuned\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.335057 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.334581 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de70e8cd-d2c4-45b4-bcd4-9a726497f781-agent-certs\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.335057 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.334711 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-tmp\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.384902 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.384554 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:29.384902 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.384586 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:29.384902 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.384599 2581 projected.go:196] Error preparing data for projected volume kube-api-access-rd59w for pod openshift-network-diagnostics/network-check-target-hdq68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:29.384902 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.384664 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w podName:af46fe6c-4921-4ca7-bb85-eec4829769af nodeName:}" failed. No retries permitted until 2026-11-03 22:22:29.884644659 +0000 UTC m=+2.391580650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rd59w" (UniqueName: "kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w") pod "network-check-target-hdq68" (UID: "af46fe6c-4921-4ca7-bb85-eec4829769af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:29.387392 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.387356 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrqs\" (UniqueName: \"kubernetes.io/projected/41972913-8aa1-4d44-97b0-033ba10133f5-kube-api-access-4jrqs\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.387508 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.387483 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpg9\" (UniqueName: \"kubernetes.io/projected/6b9b949c-4919-4d59-ae4d-e0083278cfed-kube-api-access-kcpg9\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.387554 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.387508 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv8z7\" (UniqueName: \"kubernetes.io/projected/48fecb37-0646-4895-9958-ac184f9356bd-kube-api-access-fv8z7\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:29.387554 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.387516 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9tx\" (UniqueName: \"kubernetes.io/projected/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-kube-api-access-xz9tx\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.387554 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.387535 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r484\" (UniqueName: \"kubernetes.io/projected/b6adef5f-a8df-4321-a2c3-ea90ca555c52-kube-api-access-2r484\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.387846 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.387828 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6sz\" (UniqueName: \"kubernetes.io/projected/1ac28a61-63eb-4ac5-a145-73045bf8111d-kube-api-access-7w6sz\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.388085 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.388066 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7q4p\" (UniqueName: \"kubernetes.io/projected/908bcf40-a8a2-4655-8561-473df410dfb0-kube-api-access-n7q4p\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.388164 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.388148 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dp7\" (UniqueName: \"kubernetes.io/projected/9443ca41-9780-4ecc-94c4-a1d99f138a59-kube-api-access-j2dp7\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.391777 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.391757 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hzt\" (UniqueName: \"kubernetes.io/projected/01c3b879-44a4-435d-ae16-acc5c57992b0-kube-api-access-66hzt\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.472118 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.472065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:22:29.479354 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.479336 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:22:29.517712 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.517687 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:29.536006 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.535971 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:22:29.536147 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.535981 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6rv4x" Nov 03 22:22:29.559089 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.559061 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:22:29.567819 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.567786 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:22:29.622498 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.622466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:29.628079 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.628057 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:22:29.681683 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:29.681639 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aaf8075c83069da4e0349ec07734525.slice/crio-80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22 WatchSource:0}: Error finding container 80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22: Status 404 returned error can't find the container with id 80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22 Nov 03 22:22:29.682123 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:29.682102 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79e0f53b5b82fd1a135f4636735856a.slice/crio-e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23 WatchSource:0}: Error finding container e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23: Status 404 returned error can't find the container with id e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23 Nov 03 22:22:29.687533 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.687517 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:22:29.740705 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.740677 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:29.836285 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.836249 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:29.836285 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.836291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:29.836480 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.836398 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:29.836480 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.836443 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs podName:48fecb37-0646-4895-9958-ac184f9356bd nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.83642999 +0000 UTC m=+3.343365973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs") pod "network-metrics-daemon-xfjg6" (UID: "48fecb37-0646-4895-9958-ac184f9356bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:29.836480 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.836397 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:29.836578 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.836481 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret podName:b2e63ee2-e60b-45fd-90e4-e60785022ff7 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.836474899 +0000 UTC m=+3.343410882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret") pod "global-pull-secret-syncer-zlkqx" (UID: "b2e63ee2-e60b-45fd-90e4-e60785022ff7") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:29.937647 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:29.937567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:29.938190 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.937685 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:29.938190 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.937698 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:29.938190 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.937706 2581 projected.go:196] Error preparing data for projected volume kube-api-access-rd59w for pod openshift-network-diagnostics/network-check-target-hdq68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:29.938190 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:29.937754 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w podName:af46fe6c-4921-4ca7-bb85-eec4829769af nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.937742969 +0000 UTC m=+3.444678951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd59w" (UniqueName: "kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w") pod "network-check-target-hdq68" (UID: "af46fe6c-4921-4ca7-bb85-eec4829769af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:30.057387 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.057331 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"d79e0f53b5b82fd1a135f4636735856a","Type":"ContainerStarted","Data":"e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23"} Nov 03 22:22:30.058242 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.058224 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"5aaf8075c83069da4e0349ec07734525","Type":"ContainerStarted","Data":"80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22"} Nov 03 22:22:30.168462 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.168433 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:30.247036 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.246964 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:30.289940 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.289878 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:17:29 +0000 UTC" deadline="2028-07-17 10:21:51.782237724 +0000 UTC" Nov 03 22:22:30.289940 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.289930 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14915h59m21.492309608s" Nov 03 22:22:30.368850 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.368823 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41972913_8aa1_4d44_97b0_033ba10133f5.slice/crio-57cc701c9438a0dcda7e3588f029ae67bb7358e3180f37f49fc8a581ab23aec5 WatchSource:0}: Error finding container 57cc701c9438a0dcda7e3588f029ae67bb7358e3180f37f49fc8a581ab23aec5: Status 404 returned error can't find the container with id 57cc701c9438a0dcda7e3588f029ae67bb7358e3180f37f49fc8a581ab23aec5 Nov 03 22:22:30.378774 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.378750 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c3b879_44a4_435d_ae16_acc5c57992b0.slice/crio-eb6002e7cb3e22de0b69530b64f8473e24d0b83e774b6da04ad653b6b0fef46a WatchSource:0}: Error finding container eb6002e7cb3e22de0b69530b64f8473e24d0b83e774b6da04ad653b6b0fef46a: Status 404 returned error can't find the container with id eb6002e7cb3e22de0b69530b64f8473e24d0b83e774b6da04ad653b6b0fef46a Nov 03 22:22:30.401645 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.401621 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edba00c_b9a2_4437_a117_4cbd8c7f3fc3.slice/crio-8e8f6897c1cbf9a54c059bcc859fd4ab5eeb4a0edaf13305062d06d5a4e8df22 WatchSource:0}: Error finding container 8e8f6897c1cbf9a54c059bcc859fd4ab5eeb4a0edaf13305062d06d5a4e8df22: Status 404 returned error can't find the container with id 8e8f6897c1cbf9a54c059bcc859fd4ab5eeb4a0edaf13305062d06d5a4e8df22 Nov 03 22:22:30.425814 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.425774 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9443ca41_9780_4ecc_94c4_a1d99f138a59.slice/crio-aef90eda4fe9cb75d5fc2cd2d9bd677ca110c8f9da700ab252843435cea2ab91 WatchSource:0}: Error finding container aef90eda4fe9cb75d5fc2cd2d9bd677ca110c8f9da700ab252843435cea2ab91: Status 404 returned error can't find the container with id aef90eda4fe9cb75d5fc2cd2d9bd677ca110c8f9da700ab252843435cea2ab91 Nov 03 22:22:30.454249 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.454215 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde70e8cd_d2c4_45b4_bcd4_9a726497f781.slice/crio-d2636ee5d8aeaa810525fd6a4f466d8279f60feed51558d9ae2780aad25e0ef9 WatchSource:0}: Error finding container d2636ee5d8aeaa810525fd6a4f466d8279f60feed51558d9ae2780aad25e0ef9: Status 404 returned error can't find the container with id d2636ee5d8aeaa810525fd6a4f466d8279f60feed51558d9ae2780aad25e0ef9 Nov 03 22:22:30.502931 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.502903 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9b949c_4919_4d59_ae4d_e0083278cfed.slice/crio-b638b17d190a63508913e32d555194960ebf28b204dc9522b8d5f64a4dd659e4 WatchSource:0}: Error finding container b638b17d190a63508913e32d555194960ebf28b204dc9522b8d5f64a4dd659e4: Status 404 returned error can't find the container with id b638b17d190a63508913e32d555194960ebf28b204dc9522b8d5f64a4dd659e4 Nov 03 22:22:30.503973 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.503939 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac28a61_63eb_4ac5_a145_73045bf8111d.slice/crio-4e825eb762da5ec40f3e662bc6c89e25fda32fc5733b1bc4898f3412f0ee1399 WatchSource:0}: Error finding container 4e825eb762da5ec40f3e662bc6c89e25fda32fc5733b1bc4898f3412f0ee1399: Status 404 returned error can't find the container with id 4e825eb762da5ec40f3e662bc6c89e25fda32fc5733b1bc4898f3412f0ee1399 Nov 03 22:22:30.523358 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.523333 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908bcf40_a8a2_4655_8561_473df410dfb0.slice/crio-cbc3e2c813389af4d0382b2f9b3e4debfd66d748acfa84716b73f659599764f5 WatchSource:0}: Error finding container cbc3e2c813389af4d0382b2f9b3e4debfd66d748acfa84716b73f659599764f5: Status 404 returned error can't find the container with id cbc3e2c813389af4d0382b2f9b3e4debfd66d748acfa84716b73f659599764f5 Nov 03 22:22:30.563642 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:22:30.563615 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6adef5f_a8df_4321_a2c3_ea90ca555c52.slice/crio-19283361fd7e14c1606d2adcea32f2b5be11ecd0733aaab3bda5bff9eef9955e WatchSource:0}: Error finding container 19283361fd7e14c1606d2adcea32f2b5be11ecd0733aaab3bda5bff9eef9955e: Status 404 returned error can't find the container with id 19283361fd7e14c1606d2adcea32f2b5be11ecd0733aaab3bda5bff9eef9955e Nov 03 22:22:30.845155 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.845068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:30.845155 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.845113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:30.845354 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.845225 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:30.845354 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.845227 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:30.845354 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.845282 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs podName:48fecb37-0646-4895-9958-ac184f9356bd nodeName:}" failed. No retries permitted until 2026-11-03 22:22:32.845265114 +0000 UTC m=+5.352201100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs") pod "network-metrics-daemon-xfjg6" (UID: "48fecb37-0646-4895-9958-ac184f9356bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:30.845354 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.845299 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret podName:b2e63ee2-e60b-45fd-90e4-e60785022ff7 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:32.845290945 +0000 UTC m=+5.352226931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret") pod "global-pull-secret-syncer-zlkqx" (UID: "b2e63ee2-e60b-45fd-90e4-e60785022ff7") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:30.946582 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:30.946545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:30.947255 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.946720 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:30.947255 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.946741 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:30.947255 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.946755 2581 projected.go:196] Error preparing data for projected volume kube-api-access-rd59w for pod openshift-network-diagnostics/network-check-target-hdq68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:30.947255 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:30.946835 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w podName:af46fe6c-4921-4ca7-bb85-eec4829769af nodeName:}" failed. No retries permitted until 2026-11-03 22:22:32.946815648 +0000 UTC m=+5.453751645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd59w" (UniqueName: "kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w") pod "network-check-target-hdq68" (UID: "af46fe6c-4921-4ca7-bb85-eec4829769af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:31.053261 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.053221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:31.053261 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.053248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:31.053482 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.053255 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:31.053482 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:31.053375 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:31.053561 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:31.053497 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:31.053612 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:31.053573 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:31.061559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.061518 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"cbc3e2c813389af4d0382b2f9b3e4debfd66d748acfa84716b73f659599764f5"} Nov 03 22:22:31.062727 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.062682 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerStarted","Data":"4e825eb762da5ec40f3e662bc6c89e25fda32fc5733b1bc4898f3412f0ee1399"} Nov 03 22:22:31.064632 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.064613 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" event={"ID":"6b9b949c-4919-4d59-ae4d-e0083278cfed","Type":"ContainerStarted","Data":"b638b17d190a63508913e32d555194960ebf28b204dc9522b8d5f64a4dd659e4"} Nov 03 22:22:31.065856 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.065830 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lrdk5" event={"ID":"de70e8cd-d2c4-45b4-bcd4-9a726497f781","Type":"ContainerStarted","Data":"d2636ee5d8aeaa810525fd6a4f466d8279f60feed51558d9ae2780aad25e0ef9"} Nov 03 22:22:31.067206 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.067178 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" event={"ID":"01c3b879-44a4-435d-ae16-acc5c57992b0","Type":"ContainerStarted","Data":"eb6002e7cb3e22de0b69530b64f8473e24d0b83e774b6da04ad653b6b0fef46a"} Nov 03 22:22:31.068361 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.068338 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bhsqf" event={"ID":"41972913-8aa1-4d44-97b0-033ba10133f5","Type":"ContainerStarted","Data":"57cc701c9438a0dcda7e3588f029ae67bb7358e3180f37f49fc8a581ab23aec5"} Nov 03 22:22:31.069951 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.069924 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6rv4x" event={"ID":"b6adef5f-a8df-4321-a2c3-ea90ca555c52","Type":"ContainerStarted","Data":"19283361fd7e14c1606d2adcea32f2b5be11ecd0733aaab3bda5bff9eef9955e"} Nov 03 22:22:31.071318 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.071295 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-95xrs" event={"ID":"9443ca41-9780-4ecc-94c4-a1d99f138a59","Type":"ContainerStarted","Data":"aef90eda4fe9cb75d5fc2cd2d9bd677ca110c8f9da700ab252843435cea2ab91"} Nov 03 22:22:31.072966 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.072944 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c54qw" event={"ID":"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3","Type":"ContainerStarted","Data":"8e8f6897c1cbf9a54c059bcc859fd4ab5eeb4a0edaf13305062d06d5a4e8df22"} Nov 03 22:22:31.290471 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.290369 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:17:29 +0000 UTC" deadline="2028-05-20 14:28:59.54713441 +0000 UTC" Nov 03 22:22:31.290471 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:31.290425 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13528h6m28.256713496s" Nov 03 22:22:32.871084 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:32.871006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:32.871084 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:32.871054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:32.871683 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.871188 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:32.871683 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.871254 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs podName:48fecb37-0646-4895-9958-ac184f9356bd nodeName:}" failed. No retries permitted until 2026-11-03 22:22:36.871236077 +0000 UTC m=+9.378172077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs") pod "network-metrics-daemon-xfjg6" (UID: "48fecb37-0646-4895-9958-ac184f9356bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:32.871683 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.871317 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:32.871683 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.871348 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret podName:b2e63ee2-e60b-45fd-90e4-e60785022ff7 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:36.871338053 +0000 UTC m=+9.378274049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret") pod "global-pull-secret-syncer-zlkqx" (UID: "b2e63ee2-e60b-45fd-90e4-e60785022ff7") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:32.971464 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:32.971406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:32.971675 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.971627 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:32.971675 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.971651 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:32.971675 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.971663 2581 projected.go:196] Error preparing data for projected volume kube-api-access-rd59w for pod openshift-network-diagnostics/network-check-target-hdq68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:32.971855 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:32.971721 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w podName:af46fe6c-4921-4ca7-bb85-eec4829769af nodeName:}" failed. No retries permitted until 2026-11-03 22:22:36.971703603 +0000 UTC m=+9.478639604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd59w" (UniqueName: "kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w") pod "network-check-target-hdq68" (UID: "af46fe6c-4921-4ca7-bb85-eec4829769af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:33.054157 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:33.053954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:33.054294 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:33.054270 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:33.054357 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:33.054028 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:33.054414 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:33.054382 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:33.054414 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:33.053935 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:33.054514 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:33.054458 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:33.094609 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:33.094555 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"5aaf8075c83069da4e0349ec07734525","Type":"ContainerStarted","Data":"82eca44b8f7b3c182b0e2c879baa7f4852dbfa269abc452736d616bf585551de"} Nov 03 22:22:34.104964 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:34.104144 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"d79e0f53b5b82fd1a135f4636735856a","Type":"ContainerStarted","Data":"cce60f8e5fdd9b268b6a889325976f6e88363943c4e4a95a8bec19fa56234692"} Nov 03 22:22:34.117480 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:34.117441 2581 generic.go:356] "Generic (PLEG): container finished" podID="5aaf8075c83069da4e0349ec07734525" containerID="82eca44b8f7b3c182b0e2c879baa7f4852dbfa269abc452736d616bf585551de" exitCode=0 Nov 03 22:22:34.117619 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:34.117539 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"5aaf8075c83069da4e0349ec07734525","Type":"ContainerDied","Data":"82eca44b8f7b3c182b0e2c879baa7f4852dbfa269abc452736d616bf585551de"} Nov 03 22:22:34.130311 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:34.130232 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerStarted","Data":"59e1bf32fc6e38526babda8d024e5f2c1bbed9dc8ce89c2818f1c59ba91d55db"} Nov 03 22:22:34.211497 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:34.211394 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" podStartSLOduration=6.21137345 podStartE2EDuration="6.21137345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:34.210207312 +0000 UTC m=+6.717143317" watchObservedRunningTime="2026-11-03 22:22:34.21137345 +0000 UTC m=+6.718309457" Nov 03 22:22:35.053268 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:35.053231 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:35.053438 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:35.053384 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:35.053929 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:35.053897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:35.054053 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:35.054033 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:35.054119 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:35.054098 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:35.054184 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:35.054168 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:35.141798 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:35.141752 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"5aaf8075c83069da4e0349ec07734525","Type":"ContainerStarted","Data":"9552617530a51bf16d2ec4ac3558837c036cea201a0770dbf8375c4079493afb"} Nov 03 22:22:35.144588 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:35.144561 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="59e1bf32fc6e38526babda8d024e5f2c1bbed9dc8ce89c2818f1c59ba91d55db" exitCode=0 Nov 03 22:22:35.145497 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:35.145474 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerDied","Data":"59e1bf32fc6e38526babda8d024e5f2c1bbed9dc8ce89c2818f1c59ba91d55db"} Nov 03 22:22:35.250079 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:35.250014 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" podStartSLOduration=7.249994475 podStartE2EDuration="7.249994475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:35.249349523 +0000 UTC m=+7.756285529" watchObservedRunningTime="2026-11-03 22:22:35.249994475 +0000 UTC m=+7.756930492" Nov 03 22:22:36.154366 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:36.153987 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lrdk5" event={"ID":"de70e8cd-d2c4-45b4-bcd4-9a726497f781","Type":"ContainerStarted","Data":"87114bc6b14b4b2e3e38b1f5595c9997a20226609ccb51bbe08de38dd4e9cfe7"} Nov 03 22:22:36.541491 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:36.541426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrdk5_de70e8cd-d2c4-45b4-bcd4-9a726497f781/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.895277 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:36.895112 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrdk5_de70e8cd-d2c4-45b4-bcd4-9a726497f781/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.907579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:36.906876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:36.907579 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:36.906926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:36.907579 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:36.907049 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:36.907579 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:36.907106 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs podName:48fecb37-0646-4895-9958-ac184f9356bd nodeName:}" failed. No retries permitted until 2026-11-03 22:22:44.907088426 +0000 UTC m=+17.414024431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs") pod "network-metrics-daemon-xfjg6" (UID: "48fecb37-0646-4895-9958-ac184f9356bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:36.907579 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:36.907479 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:36.907579 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:36.907527 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret podName:b2e63ee2-e60b-45fd-90e4-e60785022ff7 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:44.907512108 +0000 UTC m=+17.414448111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret") pod "global-pull-secret-syncer-zlkqx" (UID: "b2e63ee2-e60b-45fd-90e4-e60785022ff7") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:37.007291 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.007248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:37.007451 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:37.007420 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:37.007451 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:37.007438 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:37.007451 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:37.007452 2581 projected.go:196] Error preparing data for projected volume kube-api-access-rd59w for pod openshift-network-diagnostics/network-check-target-hdq68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:37.007592 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:37.007533 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w podName:af46fe6c-4921-4ca7-bb85-eec4829769af nodeName:}" failed. No retries permitted until 2026-11-03 22:22:45.007514087 +0000 UTC m=+17.514450088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd59w" (UniqueName: "kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w") pod "network-check-target-hdq68" (UID: "af46fe6c-4921-4ca7-bb85-eec4829769af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:37.053703 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.053666 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:37.054019 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:37.053828 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:37.054019 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.053893 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:37.054019 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:37.053963 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:37.054019 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.054002 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:37.054234 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:37.054060 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:37.119109 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.119077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrdk5_de70e8cd-d2c4-45b4-bcd4-9a726497f781/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.161335 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.161188 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" event={"ID":"01c3b879-44a4-435d-ae16-acc5c57992b0","Type":"ContainerStarted","Data":"c3a7e0365351b9e436b5e429ef27b96efd89898d95f53a31a8e19a4f9138ee63"} Nov 03 22:22:37.167290 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.167039 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-95xrs" event={"ID":"9443ca41-9780-4ecc-94c4-a1d99f138a59","Type":"ContainerStarted","Data":"78be54285294d12322348c47b4d6b2addfaa39ac8e2c53e8f43a8c934ac1ef10"} Nov 03 22:22:37.279769 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.278665 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lrdk5" podStartSLOduration=4.438967883 podStartE2EDuration="9.278642788s" totalImagesPullingTime="4.839674905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:36.265177526 +0000 UTC m=+8.772113532" watchObservedRunningTime="2026-11-03 22:22:37.278642788 +0000 UTC m=+9.785578798" Nov 03 22:22:37.279769 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.279635 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-95xrs" podStartSLOduration=3.601823164 podStartE2EDuration="9.279623623s" totalImagesPullingTime="5.677800459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:37.278933322 +0000 UTC m=+9.785869329" watchObservedRunningTime="2026-11-03 22:22:37.279623623 +0000 UTC m=+9.786559655" Nov 03 22:22:37.950980 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.950940 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:37.951719 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:37.951685 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:38.651207 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:38.651185 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:22:39.002010 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.001881 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:22:38.651203586Z","UUID":"2d772ae6-0e52-43d2-b310-50d3e2d26439","Handler":null,"Name":"","Endpoint":""} Nov 03 22:22:39.003829 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.003501 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:22:39.003829 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.003533 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:22:39.053660 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.053633 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:39.053660 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.053655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:39.053850 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:39.053774 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:39.053914 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:39.053840 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:39.053914 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.053862 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:39.054013 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:39.053945 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:39.173796 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.173766 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" event={"ID":"01c3b879-44a4-435d-ae16-acc5c57992b0","Type":"ContainerStarted","Data":"1b8a684ce5944ea3e126de1dcbb1de14eb5cddcf5a9ab8e866abe4b9df7de4d4"} Nov 03 22:22:39.175689 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.175665 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:22:39.176620 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:39.176596 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6rv4x" event={"ID":"b6adef5f-a8df-4321-a2c3-ea90ca555c52","Type":"ContainerStarted","Data":"f9c56538e5f42c2a15758e1ab41029c8ea3a6e4f984bfde009311531ed409a62"} Nov 03 22:22:40.180753 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:40.180514 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="02539120311a1634a97a2179d095db23a502e187b91dd61a7813ba853a31177f" exitCode=0 Nov 03 22:22:40.181248 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:40.180614 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerDied","Data":"02539120311a1634a97a2179d095db23a502e187b91dd61a7813ba853a31177f"} Nov 03 22:22:40.182516 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:40.182481 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bhsqf" event={"ID":"41972913-8aa1-4d44-97b0-033ba10133f5","Type":"ContainerStarted","Data":"98912f8d44b1c5afd24bfd9703bab8ef277b41e2fcb530a8e04976d7f93727f6"} Nov 03 22:22:40.295879 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:40.295795 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6rv4x" podStartSLOduration=4.262214573 podStartE2EDuration="12.295779429s" totalImagesPullingTime="8.033564856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:39.295623189 +0000 UTC m=+11.802559197" watchObservedRunningTime="2026-11-03 22:22:40.295779429 +0000 UTC m=+12.802715434" Nov 03 22:22:40.402596 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:40.402540 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bhsqf" podStartSLOduration=3.5528996619999997 podStartE2EDuration="12.402522137s" totalImagesPullingTime="8.849622475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:40.402223076 +0000 UTC m=+12.909159081" watchObservedRunningTime="2026-11-03 22:22:40.402522137 +0000 UTC m=+12.909458142" Nov 03 22:22:41.053925 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.053899 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:41.054032 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.053985 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:41.054124 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.054104 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:41.054178 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:41.054116 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:41.054258 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:41.054226 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:41.054357 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:41.054340 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:41.186736 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.186695 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c54qw" event={"ID":"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3","Type":"ContainerStarted","Data":"be6739fdc4981efbce1e51cc18694bf0d31535c42b2945b09e0da4b412ae2110"} Nov 03 22:22:41.188515 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.188488 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" event={"ID":"6b9b949c-4919-4d59-ae4d-e0083278cfed","Type":"ContainerStarted","Data":"1006e914bbd3be8515693e50eb70525d0c53d6cb428cf6fd2655d7ca7b2bdc05"} Nov 03 22:22:41.193429 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.193403 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" event={"ID":"01c3b879-44a4-435d-ae16-acc5c57992b0","Type":"ContainerStarted","Data":"5ff904bffd03da500635c7746a5fe568ee0009635adacbe76fc77ef15945efc0"} Nov 03 22:22:41.293004 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.292905 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c54qw" podStartSLOduration=4.680170862 podStartE2EDuration="13.292887713s" totalImagesPullingTime="8.612716851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:41.292884669 +0000 UTC m=+13.799820675" watchObservedRunningTime="2026-11-03 22:22:41.292887713 +0000 UTC m=+13.799823718" Nov 03 22:22:41.402865 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.402798 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" podStartSLOduration=2.863601377 podStartE2EDuration="13.402784959s" totalImagesPullingTime="10.539183582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:41.402700648 +0000 UTC m=+13.909636654" watchObservedRunningTime="2026-11-03 22:22:41.402784959 +0000 UTC m=+13.909720961" Nov 03 22:22:41.513096 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.513050 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" podStartSLOduration=4.102182903 podStartE2EDuration="13.513036897s" totalImagesPullingTime="9.410853994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:41.512485216 +0000 UTC m=+14.019421235" watchObservedRunningTime="2026-11-03 22:22:41.513036897 +0000 UTC m=+14.019972901" Nov 03 22:22:41.519554 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:41.519530 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-95xrs_9443ca41-9780-4ecc-94c4-a1d99f138a59/node-ca/0.log" Nov 03 22:22:42.197675 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:42.197636 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="66da736b12e23f123b4083d505d11106f2d912bc60134cd8cd0756097437ff48" exitCode=0 Nov 03 22:22:42.198398 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:42.197748 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerDied","Data":"66da736b12e23f123b4083d505d11106f2d912bc60134cd8cd0756097437ff48"} Nov 03 22:22:43.053955 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:43.053917 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:43.054119 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:43.054000 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:43.054119 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:43.054085 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:43.054232 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:43.054109 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:43.054232 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:43.054163 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:43.054332 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:43.054272 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:44.204651 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:44.204397 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="f906e0bf2c6ae84646d81d78811986d7b3c9357976dd118df30c0c9044e83913" exitCode=0 Nov 03 22:22:44.204651 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:44.204465 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerDied","Data":"f906e0bf2c6ae84646d81d78811986d7b3c9357976dd118df30c0c9044e83913"} Nov 03 22:22:44.972376 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:44.972344 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:44.972567 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:44.972386 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:44.972567 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:44.972505 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:44.972567 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:44.972509 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:44.972567 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:44.972566 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs podName:48fecb37-0646-4895-9958-ac184f9356bd nodeName:}" failed. No retries permitted until 2026-11-03 22:23:00.972548608 +0000 UTC m=+33.479484605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs") pod "network-metrics-daemon-xfjg6" (UID: "48fecb37-0646-4895-9958-ac184f9356bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:44.972751 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:44.972580 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret podName:b2e63ee2-e60b-45fd-90e4-e60785022ff7 nodeName:}" failed. No retries permitted until 2026-11-03 22:23:00.972574294 +0000 UTC m=+33.479510277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret") pod "global-pull-secret-syncer-zlkqx" (UID: "b2e63ee2-e60b-45fd-90e4-e60785022ff7") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:45.053258 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:45.053227 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:45.053400 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:45.053323 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:45.053400 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:45.053328 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:45.053464 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:45.053410 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:45.053464 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:45.053447 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:45.053524 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:45.053503 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:45.073377 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:45.073349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:45.073503 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:45.073488 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:45.073540 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:45.073510 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:45.073540 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:45.073521 2581 projected.go:196] Error preparing data for projected volume kube-api-access-rd59w for pod openshift-network-diagnostics/network-check-target-hdq68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:45.073604 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:45.073562 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w podName:af46fe6c-4921-4ca7-bb85-eec4829769af nodeName:}" failed. No retries permitted until 2026-11-03 22:23:01.073548704 +0000 UTC m=+33.580484703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd59w" (UniqueName: "kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w") pod "network-check-target-hdq68" (UID: "af46fe6c-4921-4ca7-bb85-eec4829769af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:45.208294 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:45.208265 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"26a655f7afbab3e8f441e8d2d6e82270a925015608957e02d1df2de24ce36736"} Nov 03 22:22:45.208593 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:45.208366 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"f8c2c61deecb0738a723a4b8b106d2bec3be3433c6606302bc38dfe22c57d1c5"} Nov 03 22:22:46.214504 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:46.214320 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"16e147d91ee2ca6db77da1cfd29aa2e2dcdde7b19e3591d64fb0bac202eb6964"} Nov 03 22:22:46.214893 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:46.214546 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"ae34989f7361e6abcd440521778b7daaa4f08a3b4bb6071605d20db769f33a8a"} Nov 03 22:22:46.214893 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:46.214585 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"c97aa237213a2abb8e8a596598ffef9eb67feddc138d731cbfa35b36e933058d"} Nov 03 22:22:46.214893 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:46.214615 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"393af680bbb90833d1cf006bd62de0bd3f3dc4f28bbabf29cb49fbed055bf653"} Nov 03 22:22:47.053901 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:47.053824 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:47.053901 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:47.053887 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:47.054066 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:47.053989 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:47.054066 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:47.054017 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:47.054130 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:47.054071 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:47.054174 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:47.054147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:47.218759 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:47.218725 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="b4c2b9c295ee5ada60dd0f092fad69eee7373acfc7a56fd8eb85b8b5effbf8ab" exitCode=0 Nov 03 22:22:47.219201 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:47.218777 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerDied","Data":"b4c2b9c295ee5ada60dd0f092fad69eee7373acfc7a56fd8eb85b8b5effbf8ab"} Nov 03 22:22:48.223750 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:48.223708 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"b8339d548d2a4e63ad31374fce7720d4f26399fed46e903f29b592bb159ea79f"} Nov 03 22:22:48.226653 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:48.226622 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="7d052b166edaa4212936fbeb0fba68abb03f4de8d7b0d9dcc1766f272154be4a" exitCode=0 Nov 03 22:22:48.226764 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:48.226669 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerDied","Data":"7d052b166edaa4212936fbeb0fba68abb03f4de8d7b0d9dcc1766f272154be4a"} Nov 03 22:22:49.053947 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.053914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:49.053947 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.053951 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:49.053947 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.053946 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:49.054238 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:49.054053 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:49.054238 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:49.054152 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:49.054238 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:49.054231 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:49.231953 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.231917 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x22cf" event={"ID":"1ac28a61-63eb-4ac5-a145-73045bf8111d","Type":"ContainerStarted","Data":"4844b79c0d409a75a36b9d2024bd3d0eb9c2bf98dae2b4b772f2f75c5853da5f"} Nov 03 22:22:49.704623 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.704583 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:49.704880 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.704751 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:22:49.705324 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.705298 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:22:49.820517 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:49.820048 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x22cf" podStartSLOduration=9.204148856 podStartE2EDuration="21.820032026s" totalImagesPullingTime="12.61588317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:49.360171637 +0000 UTC m=+21.867107643" watchObservedRunningTime="2026-11-03 22:22:49.820032026 +0000 UTC m=+22.326968028" Nov 03 22:22:50.237257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:50.237173 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" event={"ID":"908bcf40-a8a2-4655-8561-473df410dfb0","Type":"ContainerStarted","Data":"a3fc51b0148b818ef7c3d044eed952d238f7140b7b178f9e6a8d15c37812b4e8"} Nov 03 22:22:50.404354 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:50.404291 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" podStartSLOduration=7.846561505 podStartE2EDuration="22.404278019s" totalImagesPullingTime="14.557716514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:50.403944899 +0000 UTC m=+22.910880904" watchObservedRunningTime="2026-11-03 22:22:50.404278019 +0000 UTC m=+22.911214023" Nov 03 22:22:51.053450 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.053417 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:51.053633 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.053491 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:51.053633 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.053579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:51.053633 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:51.053584 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:51.053820 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:51.053672 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:51.053820 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:51.053753 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:51.239483 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.239456 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:51.240081 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.239544 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:51.240081 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.239591 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:51.254424 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.254403 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:51.254622 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:51.254602 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:22:53.053434 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:53.053399 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:53.053434 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:53.053436 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:53.054030 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:53.053503 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:53.054030 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:53.053523 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:53.054030 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:53.053579 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:53.054030 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:53.053638 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:55.053945 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:55.053914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:55.054412 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:55.053914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:55.054412 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:55.054037 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:55.054412 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:55.053914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:55.054412 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:55.054103 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:55.054412 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:55.054164 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:57.053290 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:57.053253 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:57.053665 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:57.053313 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:57.053665 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:57.053405 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:57.053665 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:57.053399 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:57.053665 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:57.053493 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:22:57.053665 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:57.053556 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:59.053558 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:59.053526 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:22:59.054065 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:59.053622 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:22:59.054065 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:59.053630 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:22:59.054065 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:59.053700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:22:59.054065 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:22:59.053710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:22:59.054065 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:22:59.053828 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:23:00.297501 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.297453 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlkqx"] Nov 03 22:23:00.298149 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.298128 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:00.298252 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:00.298232 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:23:00.300818 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.300777 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hdq68"] Nov 03 22:23:00.301082 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.300931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:00.301082 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:00.301038 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:23:00.301847 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.301585 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfjg6"] Nov 03 22:23:00.301847 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.301700 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:00.301847 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:00.301818 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:23:00.988918 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.988890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:00.989069 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:00.988924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:00.989069 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:00.989033 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:23:00.989143 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:00.989087 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs podName:48fecb37-0646-4895-9958-ac184f9356bd nodeName:}" failed. No retries permitted until 2026-11-03 22:23:32.989072426 +0000 UTC m=+65.496008423 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs") pod "network-metrics-daemon-xfjg6" (UID: "48fecb37-0646-4895-9958-ac184f9356bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:23:00.989143 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:00.989033 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:23:00.989210 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:00.989156 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret podName:b2e63ee2-e60b-45fd-90e4-e60785022ff7 nodeName:}" failed. No retries permitted until 2026-11-03 22:23:32.989146104 +0000 UTC m=+65.496082100 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret") pod "global-pull-secret-syncer-zlkqx" (UID: "b2e63ee2-e60b-45fd-90e4-e60785022ff7") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:23:01.089509 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:01.089474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:01.089665 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:01.089582 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:23:01.089665 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:01.089603 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:23:01.089665 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:01.089612 2581 projected.go:196] Error preparing data for projected volume kube-api-access-rd59w for pod openshift-network-diagnostics/network-check-target-hdq68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:23:01.089665 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:01.089658 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w podName:af46fe6c-4921-4ca7-bb85-eec4829769af nodeName:}" failed. No retries permitted until 2026-11-03 22:23:33.089645728 +0000 UTC m=+65.596581724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd59w" (UniqueName: "kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w") pod "network-check-target-hdq68" (UID: "af46fe6c-4921-4ca7-bb85-eec4829769af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:23:02.054257 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:02.054077 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:02.054691 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:02.054088 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:02.054691 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:02.054337 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:23:02.054691 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:02.054108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:02.054691 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:02.054502 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:23:02.054691 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:02.054575 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:23:04.053328 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:04.053293 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:04.053841 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:04.053332 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:04.053841 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:04.053306 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:04.053841 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:04.053426 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdq68" podUID="af46fe6c-4921-4ca7-bb85-eec4829769af" Nov 03 22:23:04.053841 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:04.053517 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfjg6" podUID="48fecb37-0646-4895-9958-ac184f9356bd" Nov 03 22:23:04.053841 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:04.053606 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlkqx" podUID="b2e63ee2-e60b-45fd-90e4-e60785022ff7" Nov 03 22:23:05.740359 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.740329 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeReady" Nov 03 22:23:05.740918 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.740462 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:23:05.883724 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.883685 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-72kwh"] Nov 03 22:23:05.885708 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.885692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:05.893557 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.893536 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-54cfh"] Nov 03 22:23:05.895610 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.895588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:05.950624 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.950588 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h6v68"] Nov 03 22:23:05.952698 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.952679 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:05.960767 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.960741 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72kwh"] Nov 03 22:23:05.965729 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:05.965703 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54cfh"] Nov 03 22:23:06.019297 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019214 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:23:06.019297 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019221 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:23:06.019297 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019227 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:23:06.019509 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019494 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v9lfz\"" Nov 03 22:23:06.019566 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019544 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:23:06.019566 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019551 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:23:06.019664 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019592 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:23:06.019664 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019632 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:23:06.019752 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019695 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:23:06.019752 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019709 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:23:06.019752 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019719 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vkjzf\"" Nov 03 22:23:06.019752 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.019728 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2jqrs\"" Nov 03 22:23:06.021141 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021126 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.021197 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c213277-1f38-4e9b-8e2f-50c93be267e0-crio-socket\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.021197 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcebfc-8c28-444b-92de-5459830b78f6-config-volume\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.021284 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dcebfc-8c28-444b-92de-5459830b78f6-tmp-dir\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.021284 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schhd\" (UniqueName: \"kubernetes.io/projected/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-api-access-schhd\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.021364 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021344 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7p8\" (UniqueName: \"kubernetes.io/projected/f1dcebfc-8c28-444b-92de-5459830b78f6-kube-api-access-mn7p8\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.021407 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021365 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c213277-1f38-4e9b-8e2f-50c93be267e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.021407 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c213277-1f38-4e9b-8e2f-50c93be267e0-data-volume\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.021468 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.021418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dcebfc-8c28-444b-92de-5459830b78f6-metrics-tls\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.022276 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.022256 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6v68"] Nov 03 22:23:06.054082 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.054054 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:06.054082 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.054080 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:06.054262 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.054246 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:06.119767 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.119738 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:23:06.119767 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.119762 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:23:06.120033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.120012 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:23:06.120098 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.120043 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2dn77\"" Nov 03 22:23:06.120098 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.120070 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kbfvs\"" Nov 03 22:23:06.120098 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.120080 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:23:06.120098 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.120087 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk9bw\"" Nov 03 22:23:06.122433 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c213277-1f38-4e9b-8e2f-50c93be267e0-crio-socket\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.122481 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcebfc-8c28-444b-92de-5459830b78f6-config-volume\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.122481 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122458 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dcebfc-8c28-444b-92de-5459830b78f6-tmp-dir\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.122481 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schhd\" (UniqueName: \"kubernetes.io/projected/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-api-access-schhd\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.122669 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7p8\" (UniqueName: \"kubernetes.io/projected/f1dcebfc-8c28-444b-92de-5459830b78f6-kube-api-access-mn7p8\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.122669 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122641 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c213277-1f38-4e9b-8e2f-50c93be267e0-crio-socket\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.122669 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd763789-7d6e-4c46-b360-05c03f3d112a-cert\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:06.122771 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122741 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dcebfc-8c28-444b-92de-5459830b78f6-tmp-dir\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.122771 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c213277-1f38-4e9b-8e2f-50c93be267e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.122870 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c213277-1f38-4e9b-8e2f-50c93be267e0-data-volume\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.122870 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dcebfc-8c28-444b-92de-5459830b78f6-metrics-tls\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.122963 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.123015 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.122988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84gm\" (UniqueName: \"kubernetes.io/projected/dd763789-7d6e-4c46-b360-05c03f3d112a-kube-api-access-c84gm\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:06.123158 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.123138 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcebfc-8c28-444b-92de-5459830b78f6-config-volume\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.123218 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.123197 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c213277-1f38-4e9b-8e2f-50c93be267e0-data-volume\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.123408 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.123386 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.126437 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.126412 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dcebfc-8c28-444b-92de-5459830b78f6-metrics-tls\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.126517 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.126413 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c213277-1f38-4e9b-8e2f-50c93be267e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.182656 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.182620 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-schhd\" (UniqueName: \"kubernetes.io/projected/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-api-access-schhd\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.182656 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.182621 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7p8\" (UniqueName: \"kubernetes.io/projected/f1dcebfc-8c28-444b-92de-5459830b78f6-kube-api-access-mn7p8\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.197486 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.197461 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:06.206880 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.206862 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:06.224385 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.224355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd763789-7d6e-4c46-b360-05c03f3d112a-cert\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:06.224512 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.224448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84gm\" (UniqueName: \"kubernetes.io/projected/dd763789-7d6e-4c46-b360-05c03f3d112a-kube-api-access-c84gm\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:06.226752 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.226730 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd763789-7d6e-4c46-b360-05c03f3d112a-cert\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:06.282715 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.282688 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84gm\" (UniqueName: \"kubernetes.io/projected/dd763789-7d6e-4c46-b360-05c03f3d112a-kube-api-access-c84gm\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:06.562476 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.562397 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:06.635107 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.635079 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72kwh"] Nov 03 22:23:06.635594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.635575 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54cfh"] Nov 03 22:23:06.643850 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:06.643824 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dcebfc_8c28_444b_92de_5459830b78f6.slice/crio-92cf61d0be80e7241a3bfa3a96080d37536c00b6f27f522b0025a9e92278f7e9 WatchSource:0}: Error finding container 92cf61d0be80e7241a3bfa3a96080d37536c00b6f27f522b0025a9e92278f7e9: Status 404 returned error can't find the container with id 92cf61d0be80e7241a3bfa3a96080d37536c00b6f27f522b0025a9e92278f7e9 Nov 03 22:23:06.644590 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:06.644572 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c213277_1f38_4e9b_8e2f_50c93be267e0.slice/crio-4ff8207d7c92cf76de93a5e2d9530881b1677c182839c61014f535a2538cca79 WatchSource:0}: Error finding container 4ff8207d7c92cf76de93a5e2d9530881b1677c182839c61014f535a2538cca79: Status 404 returned error can't find the container with id 4ff8207d7c92cf76de93a5e2d9530881b1677c182839c61014f535a2538cca79 Nov 03 22:23:06.925770 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:06.925661 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6v68"] Nov 03 22:23:06.933483 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:06.933460 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd763789_7d6e_4c46_b360_05c03f3d112a.slice/crio-40f83a693c2d0ba8e95542b742cd35bf99e3680d7be70679d5003674059e4bdc WatchSource:0}: Error finding container 40f83a693c2d0ba8e95542b742cd35bf99e3680d7be70679d5003674059e4bdc: Status 404 returned error can't find the container with id 40f83a693c2d0ba8e95542b742cd35bf99e3680d7be70679d5003674059e4bdc Nov 03 22:23:07.285540 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:07.284571 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6v68" event={"ID":"dd763789-7d6e-4c46-b360-05c03f3d112a","Type":"ContainerStarted","Data":"40f83a693c2d0ba8e95542b742cd35bf99e3680d7be70679d5003674059e4bdc"} Nov 03 22:23:07.286733 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:07.286703 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54cfh" event={"ID":"7c213277-1f38-4e9b-8e2f-50c93be267e0","Type":"ContainerStarted","Data":"b655f3df73f4ba86ac14377ce2b434b3f3645ee3a99c79d749d7d25ed81082a6"} Nov 03 22:23:07.286900 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:07.286764 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54cfh" event={"ID":"7c213277-1f38-4e9b-8e2f-50c93be267e0","Type":"ContainerStarted","Data":"4ff8207d7c92cf76de93a5e2d9530881b1677c182839c61014f535a2538cca79"} Nov 03 22:23:07.287837 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:07.287815 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72kwh" event={"ID":"f1dcebfc-8c28-444b-92de-5459830b78f6","Type":"ContainerStarted","Data":"92cf61d0be80e7241a3bfa3a96080d37536c00b6f27f522b0025a9e92278f7e9"} Nov 03 22:23:08.292725 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:08.292667 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54cfh" event={"ID":"7c213277-1f38-4e9b-8e2f-50c93be267e0","Type":"ContainerStarted","Data":"acf57bfcb13f883e13f5c663a4d929b84d2ed87002e12ee9dbb39aed35ace097"} Nov 03 22:23:09.298445 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:09.298359 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72kwh" event={"ID":"f1dcebfc-8c28-444b-92de-5459830b78f6","Type":"ContainerStarted","Data":"38f22e24aa3c4da04ecd38adf4485ea97143b0066642862325ecfa7203411c61"} Nov 03 22:23:09.298445 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:09.298420 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72kwh" event={"ID":"f1dcebfc-8c28-444b-92de-5459830b78f6","Type":"ContainerStarted","Data":"32907c506e82570a761a686ee1c0a0f828859a8b0feac011e657d696e78dff6e"} Nov 03 22:23:09.299023 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:09.298639 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:09.411675 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:09.411606 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-72kwh" podStartSLOduration=2.099851159 podStartE2EDuration="4.411586779s" totalImagesPullingTime="2.31173562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:09.41093869 +0000 UTC m=+41.917874696" watchObservedRunningTime="2026-11-03 22:23:09.411586779 +0000 UTC m=+41.918522784" Nov 03 22:23:10.267056 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.266902 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gcrp2"] Nov 03 22:23:10.270128 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.270108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.295136 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.295110 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb"] Nov 03 22:23:10.297090 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.297075 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.303390 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.302612 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6v68" event={"ID":"dd763789-7d6e-4c46-b360-05c03f3d112a","Type":"ContainerStarted","Data":"f57168dfd4510aa7f5e33465d0efaa6c627321a2c269ae62bb4b3bb604560f08"} Nov 03 22:23:10.306492 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.306464 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54cfh" event={"ID":"7c213277-1f38-4e9b-8e2f-50c93be267e0","Type":"ContainerStarted","Data":"4f32757c07a91d9c14e9be22ea4a68737d7c2020ae3fa2a960ae3173a16a4c3e"} Nov 03 22:23:10.321033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.320989 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:23:10.321033 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.321008 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:23:10.340538 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.340513 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:23:10.344670 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.344643 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:23:10.344772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.344684 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2kdj\"" Nov 03 22:23:10.344772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.344694 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:23:10.344772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.344693 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:23:10.354538 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.354664 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354555 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-wtmp\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.354664 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.354664 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6gn\" (UniqueName: \"kubernetes.io/projected/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-api-access-4s6gn\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.354799 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354690 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.354896 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.354966 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-sys\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.354966 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.354928 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-root\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.355091 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.355073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.355146 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.355124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-textfile\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.355215 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.355191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-tls\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.355275 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.355256 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-metrics-client-ca\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.355353 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.355336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.355391 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.355377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74p2\" (UniqueName: \"kubernetes.io/projected/838971d1-4352-4fa8-80ec-f45ceb86b53c-kube-api-access-j74p2\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.355425 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.355406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ac641f1-837b-47f2-8aaf-09d93df2be1b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.365662 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.365638 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:23:10.365863 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.365838 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f4nwd\"" Nov 03 22:23:10.365943 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.365921 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 22:23:10.366686 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.366662 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb"] Nov 03 22:23:10.419700 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.419667 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 22:23:10.456287 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.456287 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-textfile\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456515 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456306 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-tls\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456515 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-metrics-client-ca\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456515 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456515 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74p2\" (UniqueName: \"kubernetes.io/projected/838971d1-4352-4fa8-80ec-f45ceb86b53c-kube-api-access-j74p2\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456515 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ac641f1-837b-47f2-8aaf-09d93df2be1b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-wtmp\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456606 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6gn\" (UniqueName: \"kubernetes.io/projected/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-api-access-4s6gn\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456685 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-textfile\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-sys\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.456781 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-root\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.457184 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456845 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-wtmp\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.457184 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.456885 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-root\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.457184 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.457041 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-metrics-client-ca\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.457184 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.457121 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.457334 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.457217 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ac641f1-837b-47f2-8aaf-09d93df2be1b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.457334 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.457288 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-sys\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.457334 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.457325 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.457559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.457531 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.460397 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.460378 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-tls\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.460488 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.460381 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.460526 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.460496 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.460559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.460547 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.513613 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.513547 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74p2\" (UniqueName: \"kubernetes.io/projected/838971d1-4352-4fa8-80ec-f45ceb86b53c-kube-api-access-j74p2\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.513791 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.513774 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6gn\" (UniqueName: \"kubernetes.io/projected/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-api-access-4s6gn\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.590152 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.590123 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:10.600338 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.600295 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-54cfh" podStartSLOduration=2.291959124 podStartE2EDuration="5.600274171s" totalImagesPullingTime="3.308315047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:23:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:10.599198824 +0000 UTC m=+43.106134832" watchObservedRunningTime="2026-11-03 22:23:10.600274171 +0000 UTC m=+43.107210176" Nov 03 22:23:10.601793 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:10.601765 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838971d1_4352_4fa8_80ec_f45ceb86b53c.slice/crio-f337dbbdbaa4621ab587f3d59dae1c799f43000e4f45d32c8e18e8131dc71ba8 WatchSource:0}: Error finding container f337dbbdbaa4621ab587f3d59dae1c799f43000e4f45d32c8e18e8131dc71ba8: Status 404 returned error can't find the container with id f337dbbdbaa4621ab587f3d59dae1c799f43000e4f45d32c8e18e8131dc71ba8 Nov 03 22:23:10.612222 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.612205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:10.710061 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:10.709903 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h6v68" podStartSLOduration=2.413624009 podStartE2EDuration="5.709885096s" totalImagesPullingTime="3.296261087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:10.709432876 +0000 UTC m=+43.216368882" watchObservedRunningTime="2026-11-03 22:23:10.709885096 +0000 UTC m=+43.216821100" Nov 03 22:23:11.031107 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:11.031069 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb"] Nov 03 22:23:11.039752 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:11.039712 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac641f1_837b_47f2_8aaf_09d93df2be1b.slice/crio-5bf11df9abb1f56b533b89d2177d9a5638b6ac7efa801e0f28bfaef44cb6fbf0 WatchSource:0}: Error finding container 5bf11df9abb1f56b533b89d2177d9a5638b6ac7efa801e0f28bfaef44cb6fbf0: Status 404 returned error can't find the container with id 5bf11df9abb1f56b533b89d2177d9a5638b6ac7efa801e0f28bfaef44cb6fbf0 Nov 03 22:23:11.313861 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:11.313608 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcrp2" event={"ID":"838971d1-4352-4fa8-80ec-f45ceb86b53c","Type":"ContainerStarted","Data":"f337dbbdbaa4621ab587f3d59dae1c799f43000e4f45d32c8e18e8131dc71ba8"} Nov 03 22:23:11.314676 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:11.314655 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" event={"ID":"2ac641f1-837b-47f2-8aaf-09d93df2be1b","Type":"ContainerStarted","Data":"5bf11df9abb1f56b533b89d2177d9a5638b6ac7efa801e0f28bfaef44cb6fbf0"} Nov 03 22:23:12.319790 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.319706 2581 generic.go:356] "Generic (PLEG): container finished" podID="838971d1-4352-4fa8-80ec-f45ceb86b53c" containerID="82c08db1d9799a9559831b1ba23725a4c2fcd409f40464680758727d0287b7ba" exitCode=0 Nov 03 22:23:12.320215 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.319797 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcrp2" event={"ID":"838971d1-4352-4fa8-80ec-f45ceb86b53c","Type":"ContainerDied","Data":"82c08db1d9799a9559831b1ba23725a4c2fcd409f40464680758727d0287b7ba"} Nov 03 22:23:12.574051 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.573973 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:23:12.576625 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.576607 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.626760 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.626720 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 22:23:12.626760 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.626733 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 22:23:12.665783 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.665747 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 22:23:12.672334 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.672272 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 22:23:12.672334 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.672276 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 22:23:12.672334 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.672290 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 22:23:12.672604 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.672577 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 22:23:12.672604 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.672585 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mf8dj\"" Nov 03 22:23:12.672894 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.672877 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 22:23:12.674728 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.674709 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.674814 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.674751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.674885 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.674858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.674989 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.674953 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675064 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675152 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675152 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675152 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh4vb\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675249 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675249 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675318 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.675359 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.675344 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.719122 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.719088 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 22:23:12.719836 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.719798 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:23:12.776436 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.776585 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.776585 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4vb\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.776696 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.776758 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.776836 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776794 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.776988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777389 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777574 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.777772 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.777732 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.780963 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.780918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.781068 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.781016 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.781559 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.781534 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.781797 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.781760 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.781961 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.781941 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.782030 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.782012 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.782150 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.782131 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.782187 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.782148 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.782225 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.782207 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.833538 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.833516 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4vb\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:12.897474 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:12.897452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:13.325594 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.325356 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcrp2" event={"ID":"838971d1-4352-4fa8-80ec-f45ceb86b53c","Type":"ContainerStarted","Data":"44c042129476534c7b3e2d38718c28afeedf4db538a69bdabee97a2dbc4f9205"} Nov 03 22:23:13.326127 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.325657 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gcrp2" event={"ID":"838971d1-4352-4fa8-80ec-f45ceb86b53c","Type":"ContainerStarted","Data":"f75975251898debe745097c76ce866f6ad408fc652d3bd1a02bf896219a24833"} Nov 03 22:23:13.327388 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.327365 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" event={"ID":"2ac641f1-837b-47f2-8aaf-09d93df2be1b","Type":"ContainerStarted","Data":"2eb3973a7b2368769302d7c209b716f65481801ac3a579a750f4576b5c9ba5ef"} Nov 03 22:23:13.327503 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.327406 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" event={"ID":"2ac641f1-837b-47f2-8aaf-09d93df2be1b","Type":"ContainerStarted","Data":"36a6fdd9fe7bc02985f5ef4264bf3dcc931fc7fc6a5a032108f532a3eb7cd9c4"} Nov 03 22:23:13.327503 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.327424 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" event={"ID":"2ac641f1-837b-47f2-8aaf-09d93df2be1b","Type":"ContainerStarted","Data":"4ce811bfea6d4146ce502696bf541bf20a5cf5058b5097b75f508453b8a1396e"} Nov 03 22:23:13.382671 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.382642 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:23:13.390747 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:13.390720 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a79337_e1e4_40a7_a8cb_5fc3264f9a62.slice/crio-9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9 WatchSource:0}: Error finding container 9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9: Status 404 returned error can't find the container with id 9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9 Nov 03 22:23:13.436894 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.436839 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gcrp2" podStartSLOduration=1.894524208 podStartE2EDuration="3.43682531s" totalImagesPullingTime="1.459306217s" totalInitContainerRuntime="82.994885ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:13.436049925 +0000 UTC m=+45.942985931" watchObservedRunningTime="2026-11-03 22:23:13.43682531 +0000 UTC m=+45.943761309" Nov 03 22:23:13.547245 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:13.547185 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" podStartSLOduration=1.795391594 podStartE2EDuration="3.547165081s" totalImagesPullingTime="1.751773487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:13.546124418 +0000 UTC m=+46.053060422" watchObservedRunningTime="2026-11-03 22:23:13.547165081 +0000 UTC m=+46.054101086" Nov 03 22:23:14.331752 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:14.331713 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerStarted","Data":"9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9"} Nov 03 22:23:16.340628 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:16.340595 2581 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7" exitCode=0 Nov 03 22:23:16.340975 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:16.340643 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7"} Nov 03 22:23:16.850854 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:16.850817 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrdk5_de70e8cd-d2c4-45b4-bcd4-9a726497f781/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:17.282126 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:17.282049 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrdk5_de70e8cd-d2c4-45b4-bcd4-9a726497f781/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:19.309452 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:19.309426 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:19.351476 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:19.351452 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerStarted","Data":"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2"} Nov 03 22:23:19.351569 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:19.351517 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerStarted","Data":"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd"} Nov 03 22:23:20.357071 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:20.357035 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerStarted","Data":"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384"} Nov 03 22:23:20.357437 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:20.357100 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerStarted","Data":"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec"} Nov 03 22:23:20.357437 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:20.357125 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerStarted","Data":"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1"} Nov 03 22:23:21.364077 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:21.363994 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerStarted","Data":"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb"} Nov 03 22:23:21.622945 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:21.622849 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.739151511 podStartE2EDuration="9.62283379s" totalImagesPullingTime="6.866793544s" totalInitContainerRuntime="16.888735ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:21.597869782 +0000 UTC m=+54.104805787" watchObservedRunningTime="2026-11-03 22:23:21.62283379 +0000 UTC m=+54.129769792" Nov 03 22:23:23.253711 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:23.253683 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:33.051058 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.051023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:33.051467 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.051070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:33.144880 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.144845 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:23:33.145027 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.144897 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:23:33.152155 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.152127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:33.154188 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.154164 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:33.154319 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.154292 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:33.199852 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.199826 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:23:33.251292 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.251266 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:23:33.255724 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.255702 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:33.412491 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.412423 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk9bw\"" Nov 03 22:23:33.415518 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.415487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:33.419017 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.418759 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kbfvs\"" Nov 03 22:23:33.421857 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.421845 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:33.424441 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.424423 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2dn77\"" Nov 03 22:23:33.427586 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.427563 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:33.794957 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.794926 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlkqx"] Nov 03 22:23:33.803337 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:33.803312 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e63ee2_e60b_45fd_90e4_e60785022ff7.slice/crio-6944e293409bbb472dbc1bf7fa7777f7664b4d280daedc3535aa061e7f5e6889 WatchSource:0}: Error finding container 6944e293409bbb472dbc1bf7fa7777f7664b4d280daedc3535aa061e7f5e6889: Status 404 returned error can't find the container with id 6944e293409bbb472dbc1bf7fa7777f7664b4d280daedc3535aa061e7f5e6889 Nov 03 22:23:33.846783 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.846744 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hdq68"] Nov 03 22:23:33.847623 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:33.847595 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfjg6"] Nov 03 22:23:33.854616 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:33.854584 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fecb37_0646_4895_9958_ac184f9356bd.slice/crio-0a2881635a11f5415e17d9be84d351cc23675fd5cc413671a7c4997502fe26c9 WatchSource:0}: Error finding container 0a2881635a11f5415e17d9be84d351cc23675fd5cc413671a7c4997502fe26c9: Status 404 returned error can't find the container with id 0a2881635a11f5415e17d9be84d351cc23675fd5cc413671a7c4997502fe26c9 Nov 03 22:23:33.855198 ip-10-0-128-222 kubenswrapper[2581]: W1103 22:23:33.855177 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf46fe6c_4921_4ca7_bb85_eec4829769af.slice/crio-ab587ad5be1dbec779f642ff6a96be22cc156b4ad24b1fa917e54f63c50ec3ce WatchSource:0}: Error finding container ab587ad5be1dbec779f642ff6a96be22cc156b4ad24b1fa917e54f63c50ec3ce: Status 404 returned error can't find the container with id ab587ad5be1dbec779f642ff6a96be22cc156b4ad24b1fa917e54f63c50ec3ce Nov 03 22:23:34.402718 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:34.402685 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zlkqx" event={"ID":"b2e63ee2-e60b-45fd-90e4-e60785022ff7","Type":"ContainerStarted","Data":"6944e293409bbb472dbc1bf7fa7777f7664b4d280daedc3535aa061e7f5e6889"} Nov 03 22:23:34.403819 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:34.403788 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hdq68" event={"ID":"af46fe6c-4921-4ca7-bb85-eec4829769af","Type":"ContainerStarted","Data":"ab587ad5be1dbec779f642ff6a96be22cc156b4ad24b1fa917e54f63c50ec3ce"} Nov 03 22:23:34.404848 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:34.404829 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfjg6" event={"ID":"48fecb37-0646-4895-9958-ac184f9356bd","Type":"ContainerStarted","Data":"0a2881635a11f5415e17d9be84d351cc23675fd5cc413671a7c4997502fe26c9"} Nov 03 22:23:36.241884 ip-10-0-128-222 kubenswrapper[2581]: E1103 22:23:36.241795 2581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:23:36.416590 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:36.416552 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfjg6" event={"ID":"48fecb37-0646-4895-9958-ac184f9356bd","Type":"ContainerStarted","Data":"4c56b210529ac6978bdd62213eb169a8992c7a88f51fafb36e36d9b58314a835"} Nov 03 22:23:36.416766 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:36.416619 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfjg6" event={"ID":"48fecb37-0646-4895-9958-ac184f9356bd","Type":"ContainerStarted","Data":"03703241cd7775e219a2587558f54e39e016f2fa08233ba55ed6d6849b53364a"} Nov 03 22:23:36.530546 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:36.530435 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xfjg6" podStartSLOduration=66.832512518 podStartE2EDuration="1m8.530420897s" totalImagesPullingTime="1.697908379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:36.529865903 +0000 UTC m=+69.036801908" watchObservedRunningTime="2026-11-03 22:23:36.530420897 +0000 UTC m=+69.037356901" Nov 03 22:23:38.426226 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:38.426186 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hdq68" event={"ID":"af46fe6c-4921-4ca7-bb85-eec4829769af","Type":"ContainerStarted","Data":"5ae7eaec4ef887daaaa6099b0bf718a652c27fc5128492e2bf464b4cc2bdbc1b"} Nov 03 22:23:38.426645 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:38.426312 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:38.533402 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:38.533344 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hdq68" podStartSLOduration=66.665042342 podStartE2EDuration="1m10.533327251s" totalImagesPullingTime="3.868284909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:38.533028481 +0000 UTC m=+71.039964489" watchObservedRunningTime="2026-11-03 22:23:38.533327251 +0000 UTC m=+71.040263258" Nov 03 22:23:39.307834 ip-10-0-128-222 kubenswrapper[2581]: I1103 22:23:39.307787 2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:23:39.307990 ip-10-0-128-222 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:23:39.323132 ip-10-0-128-222 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:23:39.323296 ip-10-0-128-222 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:23:39.323469 ip-10-0-128-222 systemd[1]: kubelet.service: Consumed 3.626s CPU time. Nov 03 22:23:39.336547 ip-10-0-128-222 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:23:39.408060 ip-10-0-128-222 kubenswrapper[5272]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:23:39.408060 ip-10-0-128-222 kubenswrapper[5272]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:23:39.408060 ip-10-0-128-222 kubenswrapper[5272]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:23:39.408060 ip-10-0-128-222 kubenswrapper[5272]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:23:39.410079 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410000 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.410079 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410078 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.410079 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410083 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410086 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410090 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410094 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410097 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410100 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410103 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410105 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410108 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410111 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410113 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410117 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410125 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410128 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410131 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410134 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410137 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410140 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410143 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410146 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.410177 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410148 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410151 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410154 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410156 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410159 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410162 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410165 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410167 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410170 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410173 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410176 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410178 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410181 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410184 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410187 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410190 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410193 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410196 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410199 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410201 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.410726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410204 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410206 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410209 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410212 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410215 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410223 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410226 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410229 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410231 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410234 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410237 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410239 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410242 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410245 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410247 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410250 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410253 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410256 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410258 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410261 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.411247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410263 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410266 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410270 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410272 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410275 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410278 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410280 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410283 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410285 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410288 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410302 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410324 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410328 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410331 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410333 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410337 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410341 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410345 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410353 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.411741 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410356 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410359 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410362 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410365 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410367 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410370 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410373 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410376 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410379 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410382 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410384 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410387 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410389 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410392 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410395 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410397 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410400 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410403 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410405 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410408 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.412253 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410410 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410413 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410416 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410418 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.410424 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411217 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411225 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411229 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411234 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411237 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411240 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411243 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411247 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411250 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411253 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411256 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411258 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411261 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411265 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.412757 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411268 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411270 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411273 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411276 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411279 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411281 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411285 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411287 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411290 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411294 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411296 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411299 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411301 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411304 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411307 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411310 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411313 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411315 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411318 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411321 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.413256 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411324 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411326 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411329 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411332 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411335 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411337 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411341 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411344 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411346 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411349 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411352 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411355 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411357 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411360 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411363 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411366 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411369 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411371 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411374 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411376 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.413770 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411379 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411382 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411384 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411387 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411390 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411392 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411395 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411398 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411400 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411403 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411406 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411408 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411411 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411414 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411417 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411419 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411422 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411424 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411427 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411430 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.414278 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411433 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411435 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411438 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411441 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411443 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411446 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411449 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411451 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411454 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411457 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411459 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411462 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411465 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411467 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411470 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411472 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411475 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411478 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411481 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411484 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.414774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411487 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411489 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411492 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411495 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411499 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411504 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411508 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411511 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411514 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411517 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411520 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.411522 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412041 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412048 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412051 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412054 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412058 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412060 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412063 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.415280 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412066 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412069 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412072 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412076 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412080 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412083 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412086 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412089 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412091 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412094 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412097 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412104 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412107 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412109 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412112 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412115 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412117 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412120 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412123 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.415774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412126 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412129 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412132 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412134 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412137 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412140 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412143 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412146 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412148 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412154 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412156 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412159 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412162 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412165 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412168 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412170 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412173 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412176 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412178 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412181 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.416284 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412184 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412186 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412189 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412191 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412195 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412198 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412201 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412203 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412206 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412209 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412211 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412214 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412217 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412219 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412223 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412227 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412230 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412233 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412236 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.416781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412240 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412243 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412250 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412253 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412256 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412260 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412263 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412266 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412269 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412273 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412276 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412279 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412282 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412284 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412287 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412290 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412292 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412296 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412299 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412302 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.417267 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412304 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412307 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412310 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412312 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412316 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412319 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412328 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412331 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412334 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412336 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412339 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412342 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412345 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412348 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412353 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412356 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412359 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412361 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412364 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.417759 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412366 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.412369 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412468 5272 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412479 5272 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412488 5272 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412494 5272 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412499 5272 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412502 5272 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412506 5272 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412510 5272 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412515 5272 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412518 5272 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412522 5272 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412526 5272 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412529 5272 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412532 5272 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412535 5272 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412538 5272 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412541 5272 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412544 5272 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412550 5272 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412552 5272 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412556 5272 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412559 5272 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:23:39.418252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412563 5272 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412567 5272 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412570 5272 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412575 5272 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412579 5272 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412582 5272 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412585 5272 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412588 5272 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412591 5272 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412594 5272 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412598 5272 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412602 5272 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412604 5272 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412607 5272 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412611 5272 flags.go:64] FLAG: --enable-server="true" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412614 5272 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412618 5272 flags.go:64] FLAG: --event-burst="100" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412621 5272 flags.go:64] FLAG: --event-qps="50" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412626 5272 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412629 5272 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412633 5272 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412637 5272 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412640 5272 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412643 5272 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412646 5272 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:23:39.419227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412650 5272 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412653 5272 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412656 5272 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412659 5272 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412662 5272 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412665 5272 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412668 5272 flags.go:64] FLAG: --feature-gates="" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412672 5272 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412675 5272 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412678 5272 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412683 5272 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412686 5272 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412690 5272 flags.go:64] FLAG: --help="false" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412693 5272 flags.go:64] FLAG: --hostname-override="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412697 5272 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412700 5272 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412703 5272 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412707 5272 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412710 5272 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412713 5272 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412716 5272 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412719 5272 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412722 5272 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:23:39.419911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412725 5272 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412728 5272 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412733 5272 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412736 5272 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412739 5272 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412742 5272 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412745 5272 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412748 5272 flags.go:64] FLAG: --lock-file="" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412751 5272 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412754 5272 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412757 5272 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412763 5272 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412766 5272 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412769 5272 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412772 5272 flags.go:64] FLAG: --logging-format="text" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412775 5272 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412779 5272 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412782 5272 flags.go:64] FLAG: --manifest-url="" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412785 5272 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412790 5272 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412794 5272 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412798 5272 flags.go:64] FLAG: --max-pods="110" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412813 5272 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412816 5272 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412819 5272 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:23:39.420522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412822 5272 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412826 5272 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412830 5272 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412833 5272 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412840 5272 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412843 5272 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412846 5272 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412849 5272 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412852 5272 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412856 5272 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412859 5272 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412862 5272 flags.go:64] FLAG: --port="10250" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412865 5272 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412868 5272 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0012edbe060cd8f9c" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412871 5272 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412874 5272 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412877 5272 flags.go:64] FLAG: --register-node="true" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412880 5272 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412884 5272 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412887 5272 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412890 5272 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412892 5272 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412896 5272 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412899 5272 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412902 5272 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412905 5272 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:23:39.421163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412910 5272 flags.go:64] FLAG: --runonce="false" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412913 5272 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412916 5272 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412920 5272 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412923 5272 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412926 5272 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412929 5272 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412932 5272 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412935 5272 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412938 5272 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412941 5272 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412944 5272 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412947 5272 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412950 5272 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412953 5272 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412956 5272 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412962 5272 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412965 5272 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412968 5272 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412972 5272 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412975 5272 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412978 5272 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412981 5272 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412984 5272 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412987 5272 flags.go:64] FLAG: --v="2" Nov 03 22:23:39.421837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412991 5272 flags.go:64] FLAG: --version="false" Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412995 5272 flags.go:64] FLAG: --vmodule="" Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.412999 5272 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.413002 5272 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413132 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413136 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413139 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413142 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413145 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413148 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413150 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413154 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413157 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413160 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413162 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413165 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413168 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413170 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413173 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413175 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.422451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413178 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413181 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413183 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413186 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413188 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413191 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413194 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413197 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413199 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413202 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413205 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413207 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413210 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413213 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413215 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413218 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413220 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413223 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413226 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413229 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.422995 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413232 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413234 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413237 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413240 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413243 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413245 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413248 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413251 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413253 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413256 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413259 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413261 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413264 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413267 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413270 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413272 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413276 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413282 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413285 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413288 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.423665 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413291 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413293 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413296 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413299 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413301 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413305 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413307 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413310 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413313 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413315 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413320 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413323 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413325 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413328 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413330 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413333 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413336 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413339 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413341 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413344 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.424336 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413347 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413350 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413352 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413355 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413358 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413361 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413363 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413366 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413368 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413373 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413376 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413378 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413382 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413386 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413389 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413392 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413395 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413398 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413401 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.425818 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413403 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413407 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413409 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413413 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413416 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413419 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413421 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413424 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413427 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413429 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413432 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.413459 5272 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413494 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413498 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413501 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413504 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.426489 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413507 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413510 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413513 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413515 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413518 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413521 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413523 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413528 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413531 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413534 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413537 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413539 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413542 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413546 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413550 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413554 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413557 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413560 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413562 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413566 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.426929 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413570 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413572 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413575 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413578 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413580 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413583 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413586 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413588 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413591 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413594 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413597 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413599 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413602 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413604 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413607 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413610 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413612 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413615 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413618 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413621 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.427422 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413624 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413628 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413631 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413633 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413636 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413638 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413641 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413644 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413647 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413650 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413654 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413658 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413661 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413664 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413666 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413669 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413671 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413674 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413677 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.427928 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413679 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413682 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413685 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413688 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413691 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413693 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413696 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413698 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413701 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413703 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413706 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413709 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413713 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413715 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413718 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413721 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413724 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413726 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413729 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.428417 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413732 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413734 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413737 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413739 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413742 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413746 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413749 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413751 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413754 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413757 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413760 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413762 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413765 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413768 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413770 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413773 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413776 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413779 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413782 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413785 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.428997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413787 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413790 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413793 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.413797 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.417366 5272 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.417380 5272 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.417398 5272 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.417402 5272 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417457 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417463 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417467 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417470 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417473 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417476 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417478 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417481 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417484 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417486 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417489 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.429688 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417492 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417495 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417497 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417500 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417503 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417506 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417509 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417512 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417515 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417518 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417522 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417526 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417529 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417532 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417535 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417538 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417541 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417544 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417546 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.430475 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417549 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417552 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417555 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417558 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417560 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417563 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417566 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417569 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417571 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417574 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417576 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417579 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417582 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417585 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417588 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417591 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417593 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417597 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417600 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.431053 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417603 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417607 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417611 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417615 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417619 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417621 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417624 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417627 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417629 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417632 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417648 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417652 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417656 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417659 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417662 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417665 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417669 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417672 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417675 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417677 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.431515 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417680 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417683 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417686 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417688 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417691 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417693 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417696 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417699 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417701 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417704 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417707 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417710 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417712 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417715 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417718 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417720 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417723 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417726 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417729 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417731 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.432010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417734 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417737 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417739 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417742 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417745 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417747 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417750 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417753 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417756 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417759 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417762 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417764 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417767 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417770 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417772 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417775 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.432496 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417778 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.417820 5272 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417964 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417968 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417971 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417974 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417977 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417980 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417983 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417985 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417988 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417991 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417993 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417996 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.417999 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418002 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.432897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418004 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418007 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418010 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418013 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418015 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418018 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418020 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418023 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418026 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418028 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418031 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418033 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418036 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418038 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418041 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418043 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418046 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418049 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418051 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418054 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.433293 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418057 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418059 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418062 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418064 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418067 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418070 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418072 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418075 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418078 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418081 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418083 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418089 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418091 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418094 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418096 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418099 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418102 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418105 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418107 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418110 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.433774 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418113 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418117 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418121 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418124 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418126 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418130 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418132 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418135 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418137 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418140 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418143 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418145 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418148 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418150 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418155 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418158 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418161 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418163 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418166 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.434386 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418170 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418173 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418176 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418179 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418182 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418218 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418252 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418260 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418312 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418317 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418323 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418328 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418332 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418337 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418342 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418346 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418351 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418355 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418360 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418369 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.435010 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418374 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418378 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418382 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418387 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418391 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418396 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418400 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418407 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418411 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418416 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418420 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418424 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418434 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.418486 5272 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418613 5272 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:23:39.435528 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418620 5272 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418625 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418630 5272 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418636 5272 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418641 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418649 5272 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418654 5272 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418659 5272 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418663 5272 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418667 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418671 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418681 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418685 5272 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418690 5272 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418694 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418698 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418702 5272 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418707 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418711 5272 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418716 5272 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:23:39.435964 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418720 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418724 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418729 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418738 5272 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418745 5272 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418750 5272 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418754 5272 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418758 5272 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418762 5272 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418766 5272 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418770 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418775 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418779 5272 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418783 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418789 5272 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418793 5272 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418819 5272 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418826 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418830 5272 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418835 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:23:39.436767 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418840 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418844 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418848 5272 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418853 5272 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418858 5272 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418862 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418867 5272 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418871 5272 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418883 5272 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418890 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418895 5272 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418901 5272 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418907 5272 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418911 5272 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418916 5272 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418921 5272 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418927 5272 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418931 5272 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418936 5272 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:23:39.437621 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418940 5272 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418945 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418954 5272 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418958 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418963 5272 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418968 5272 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418973 5272 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418979 5272 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418985 5272 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418992 5272 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.418999 5272 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419004 5272 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419009 5272 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419013 5272 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419022 5272 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419027 5272 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419031 5272 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419036 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419041 5272 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:23:39.438286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419045 5272 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419050 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419054 5272 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419058 5272 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419063 5272 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419067 5272 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419071 5272 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419080 5272 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419084 5272 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419088 5272 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419094 5272 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419098 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419103 5272 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419108 5272 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419112 5272 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419116 5272 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419120 5272 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419124 5272 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419129 5272 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:23:39.438847 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419133 5272 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419143 5272 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419147 5272 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419152 5272 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419158 5272 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419164 5272 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419168 5272 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:23:39.419175 5272 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.419322 5272 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.420377 5272 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.420427 5272 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.420610 5272 server.go:1014] "Starting client certificate rotation" Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.420745 5272 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.420784 5272 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:17:28 +0000 UTC" deadline="2028-08-14 20:22:13.213964402 +0000 UTC" Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.420835 5272 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15597h58m33.793130962s" Nov 03 22:23:39.439381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.421069 5272 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:23:39.439750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.422155 5272 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:23:39.439750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.423578 5272 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:23:39.439750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.425282 5272 log.go:25] "Validated CRI v1 image API" Nov 03 22:23:39.439750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.426010 5272 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:23:39.439750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.427920 5272 fs.go:135] Filesystem UUIDs: map[03cd8e91-a896-4fdd-b375-c063837d526c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9171a9d5-de88-4fd4-b7eb-49e00613354e:/dev/nvme0n1p4] Nov 03 22:23:39.439918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.427940 5272 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a2881635a11f5415e17d9be84d351cc23675fd5cc413671a7c4997502fe26c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a2881635a11f5415e17d9be84d351cc23675fd5cc413671a7c4997502fe26c9/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19283361fd7e14c1606d2adcea32f2b5be11ecd0733aaab3bda5bff9eef9955e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19283361fd7e14c1606d2adcea32f2b5be11ecd0733aaab3bda5bff9eef9955e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40f83a693c2d0ba8e95542b742cd35bf99e3680d7be70679d5003674059e4bdc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40f83a693c2d0ba8e95542b742cd35bf99e3680d7be70679d5003674059e4bdc/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e825eb762da5ec40f3e662bc6c89e25fda32fc5733b1bc4898f3412f0ee1399/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e825eb762da5ec40f3e662bc6c89e25fda32fc5733b1bc4898f3412f0ee1399/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ff8207d7c92cf76de93a5e2d9530881b1677c182839c61014f535a2538cca79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ff8207d7c92cf76de93a5e2d9530881b1677c182839c61014f535a2538cca79/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57cc701c9438a0dcda7e3588f029ae67bb7358e3180f37f49fc8a581ab23aec5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57cc701c9438a0dcda7e3588f029ae67bb7358e3180f37f49fc8a581ab23aec5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bf11df9abb1f56b533b89d2177d9a5638b6ac7efa801e0f28bfaef44cb6fbf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bf11df9abb1f56b533b89d2177d9a5638b6ac7efa801e0f28bfaef44cb6fbf0/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6944e293409bbb472dbc1bf7fa7777f7664b4d280daedc3535aa061e7f5e6889/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6944e293409bbb472dbc1bf7fa7777f7664b4d280daedc3535aa061e7f5e6889/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e8f6897c1cbf9a54c059bcc859fd4ab5eeb4a0edaf13305062d06d5a4e8df22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e8f6897c1cbf9a54c059bcc859fd4ab5eeb4a0edaf13305062d06d5a4e8df22/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92cf61d0be80e7241a3bfa3a96080d37536c00b6f27f522b0025a9e92278f7e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92cf61d0be80e7241a3bfa3a96080d37536c00b6f27f522b0025a9e92278f7e9/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab587ad5be1dbec779f642ff6a96be22cc156b4ad24b1fa917e54f63c50ec3ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab587ad5be1dbec779f642ff6a96be22cc156b4ad24b1fa917e54f63c50ec3ce/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aef90eda4fe9cb75d5fc2cd2d9bd677ca110c8f9da700ab252843435cea2ab91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aef90eda4fe9cb75d5fc2cd2d9bd677ca110c8f9da700ab252843435cea2ab91/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbc3e2c813389af4d0382b2f9b3e4debfd66d748acfa84716b73f659599764f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbc3e2c813389af4d0382b2f9b3e4debfd66d748acfa84716b73f659599764f5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2636ee5d8aeaa810525fd6a4f466d8279f60feed51558d9ae2780aad25e0ef9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2636ee5d8aeaa810525fd6a4f466d8279f60feed51558d9ae2780aad25e0ef9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb6002e7cb3e22de0b69530b64f8473e24d0b83e774b6da04ad653b6b0fef46a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb6002e7cb3e22de0b69530b64f8473e24d0b83e774b6da04ad653b6b0fef46a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f337dbbdbaa4621ab587f3d59dae1c799f43000e4f45d32c8e18e8131dc71ba8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f337dbbdbaa4621ab587f3d59dae1c799f43000e4f45d32c8e18e8131dc71ba8/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01c3b879-44a4-435d-ae16-acc5c57992b0/volumes/kubernetes.io~projected/kube-api-access-66hzt:{mountpoint:/var/lib/kubelet/pods/01c3b879-44a4-435d-ae16-acc5c57992b0/volumes/kubernetes.io~projected/kube-api-access-66hzt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ac28a61-63eb-4ac5-a145-73045bf8111d/volumes/kubernetes.io~projected/kube-api-access-7w6sz:{mountpoint:/var/lib/kubelet/pods/1ac28a61-63eb-4ac5-a145-73045bf8111d/volumes/kubernetes.io~projected/kube-api-access-7w6sz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~projected/kube-api-access-4s6gn:{mountpoint:/var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~projected/kube-api-access-4s6gn major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41972913-8aa1-4d44-97b0-033ba10133f5/volumes/kubernetes.io~projected/kube-api-access-4jrqs:{mountpoint:/var/lib/kubelet/pods/41972913-8aa1-4d44-97b0-033ba10133f5/volumes/kubernetes.io~projected/kube-api-access-4jrqs major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48fecb37-0646-4895-9958-ac184f9356bd/volumes/kubernetes.io~projected/kube-api-access-fv8z7:{mountpoint:/var/lib/kubelet/pods/48fecb37-0646-4895-9958-ac184f9356bd/volumes/kubernetes.io~projected/kube-api-access-fv8z7 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48fecb37-0646-4895-9958-ac184f9356bd/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/48fecb37-0646-4895-9958-ac184f9356bd/volumes/kubernetes.io~secret/metrics-certs major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~projected/kube-api-access-kcpg9:{mountpoint:/var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~projected/kube-api-access-kcpg9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c213277-1f38-4e9b-8e2f-50c93be267e0/volumes/kubernetes.io~projected/kube-api-access-schhd:{mountpoint:/var/lib/kubelet/pods/7c213277-1f38-4e9b-8e2f-50c93be267e0/volumes/kubernetes.io~projected/kube-api-access-schhd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c213277-1f38-4e9b-8e2f-50c93be267e0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7c213277-1f38-4e9b-8e2f-50c93be267e0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~projected/kube-api-access-j74p2:{mountpoint:/var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~projected/kube-api-access-j74p2 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3/volumes/kubernetes.io~projected/kube-api-access-xz9tx:{mountpoint:/var/lib/kubelet/pods/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3/volumes/kubernetes.io~projected/kube-api-access-xz9tx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volumes/kubernetes.io~projected/kube-api-access-n7q4p:{mountpoint:/var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volumes/kubernetes.io~projected/kube-api-access-n7q4p major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9443ca41-9780-4ecc-94c4-a1d99f138a59/volumes/kubernetes.io~projected/kube-api-access-j2dp7:{mountpoint:/var/lib/kubelet/pods/9443ca41-9780-4ecc-94c4-a1d99f138a59/volumes/kubernetes.io~projected/kube-api-access-j2dp7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af46fe6c-4921-4ca7-bb85-eec4829769af/volumes/kubernetes.io~projected/kube-api-access-rd59w:{mountpoint:/var/lib/kubelet/pods/af46fe6c-4921-4ca7-bb85-eec4829769af/volumes/kubernetes.io~projected/kube-api-access-rd59w major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2e63ee2-e60b-45fd-90e4-e60785022ff7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b2e63ee2-e60b-45fd-90e4-e60785022ff7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6adef5f-a8df-4321-a2c3-ea90ca555c52/volumes/kubernetes.io~projected/kube-api-access-2r484:{mountpoint:/var/lib/kubelet/pods/b6adef5f-a8df-4321-a2c3-ea90ca555c52/volumes/kubernetes.io~projected/kube-api-access-2r484 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/alertmanager/9 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/config-reloader/6 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~empty-dir/config-out major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~projected/kube-api-access-jh4vb:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~projected/kube-api-access-jh4vb major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~projected/tls-assets major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/config-volume major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/web-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd763789-7d6e-4c46-b360-05c03f3d112a/volumes/kubernetes.io~projected/kube-api-access-c84gm:{mountpoint:/var/lib/kubelet/pods/dd763789-7d6e-4c46-b360-05c03f3d112a/volumes/kubernetes.io~projected/kube-api-access-c84gm major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd763789-7d6e-4c46-b360-05c03f3d112a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/dd763789-7d6e-4c46-b360-05c03f3d112a/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de70e8cd-d2c4-45b4-bcd4-9a726497f781/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/de70e8cd-d2c4-45b4-bcd4-9a726497f781/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1dcebfc-8c28-444b-92de-5459830b78f6/volumes/kubernetes.io~projected/kube-api-access-mn7p8:{mountpoint:/var/lib/kubelet/pods/f1dcebfc-8c28-444b-92de-5459830b78f6/volumes/kubernetes.io~projected/kube-api-access-mn7p8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1dcebfc-8c28-444b-92de-5459830b78f6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f1dcebfc-8c28-444b-92de-5459830b78f6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2dd35c735291a3423081e923d7b640b3aa100fdb3059dba30e2e2f5926b9007c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/44bac2467eb958bdb275aed873ffa198a4061186e8276db76fd26c4c9f4cfbcd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4406d03b829c2854a360e1b2b8b94a2bde4f8084ae2b1c95e134149bdced21e9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/740dd0b3e1ecac10826f200fb2dce51a3f10d2ac35a3b1dd2d40441f26a1a5b8/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0f3028ce6fcc2d47695b465f12828af1e71dd949f8a5676f6fdfbc7418b30e9d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/60f56a51f1bb6494996ba8b3f5fcf19fb46e5172cfe6775c1e9a01b797b9b5ab/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e59950ef3c2d68bd6cc9825d593f470335d94505b8c27e8ad54d399b9703e9b4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/627b2aae078a80a3956070bd49e6adf7d16874db106256a98613b94658551a7e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/55ea8cabe20565d281e7e51334c1693aa1bdefd7aac7ee7f8b6799c763ad1c33/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6077bdf756f6903b803a063a32ccd3d68b6cf1e507d2ea9b1b226c2e394a7901/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e28b22d5f83c466c13da20a4ff496278542c4ea0ec2509ce9e1b866ac1af5d83/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/325ba7c9356fca7674ece158d7df4e18794322234cd872f6d1adbe92a06bef56/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5353f4a82c2a74e6fd0b86d2c0a9b724ac0bcf8d0d46e97adbb282589e6c0a9a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1bc43c9dfb60ac20c7a5e5895631982c8c09e842f557ea62ebd655b7a76c4db4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a0e5b00ed70c1f7d91bdd6ca9ef0e0555cd3185b2bd1d984a650ab8f2a94995b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/493f0744c0fb1de53bb3dba3653dedd9de5b2b7bcc0e8ba1523c5118515eef86/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1772770370ec1f1d3b9a2fc0bb5d230c269d52c868a5dab56361d166d3f9bce8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/49333d958e292b64a6b50bc0f51612b6fae4d0b0b569432e0af89b79182fbe83/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/23f77e2ebebb090a9c14800e240dab3353ece058a9f85d8324095f12af15bbea/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/43bd44eaf050028ece568d9a30f720a45fe6b2a6aaeaed7d0a0e4a71a927a89b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ce2eeb035df83c2a923ef74d9eee6d7209a505ba1731ba7819ead3f240041f10/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/fb1ad54be5369bb99467c129a36699508b7d5ca8ca527ac1214d57ad67611658/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2ab45e96aa77346009fd3a1ad94342203e1793af96726f0f41a2144e953ddbdd/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/7b7b2bdb5e26d7990a1eee9ee4ae55c1ccb9c2c63eff5fb75fd05c51bf3e2064/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/cb9a9fdf2400a58f75b7663d762d090b80b505cbec58526e5fbb243a4c96b3ab/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/ac013a0edb13a5a79fc8af2f081203e56fcc644a0253cf3f9f110ba22e10b081/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/fe10baf2b44ab92e9db5e3d09d4a7243dabeeb8a979cf2ecc85faaefdad90b32/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/51e27a3fecc34bca953fe3313fa799b73c9a05666a264c764d9c215629d25d95/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/9edf6ee96edb6ccd590987ea7b7b5eda668b0cc94518a6182163e652cdd040cf/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/20d23d3379c3697b25bc4804aa5cc6e9b049162b3710a1369888aba0c5940fe8/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/0028668c6ddf19f8cfa3feb2c9072ff9e33e534310251d3651223b289b07f235/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/909406d71fbfaa133610a069c2ac32dc1b309892500a89e22d3a749027a90428/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5b4754b65e83b858d49268fd5b9c8a47195c0596578aec135a1248748c873ca6/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/60f7180f3d57c609d1892373abf6552a27c4b6ee9596096a620fd2b47170f800/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/1ebd3ef0ab7987972a8e30cd87321ddb6c757ae373ed8d431973a8cb53405995/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/7a6c795083f5622514819f10518549ce7f86f3eea53ba33afdf7705add40ebb2/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/8c7cfea86b263b3048b29d893361498faf3b28e4aad87a4f40e6876eff051126/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/14966f75434867473b2bf60f3cb0e605b590891489956e335818be9d45cb9fa3/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/14eee59e514a3106c242ee3221522d1e7d9a31c75c1ec0886ab0ea5cbd3d529b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/5e221bfcd5e121a33152e2cb8e3f8013d05b85fd9b539f39fd0a0fcef9f42f11/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/94a10f0bea81df678f54e2a5bc083ca73bb5aa5b8f7e8b38bb0114c5fbca457c/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/bc486b1be1369942bedea6ed4a3ef3fd2172bd0af850d9a313c87730131cf734/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a7033ea772a2bcdb925e51d06cbd9e87018fae6be9ff63a7d4707c2a5a9e3ac4/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/9beadaefb1f9fbcbb229f4d7fb476b629aa1b3742d3a92d1ea972a1381bd14a6/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/f0d0e6ea04f947427cc4c97de15227d3008a1d9b00149ad9ebcdfc1ac01a9667/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/5f45d77717c2d6ec955be5bd0ad02a3563adab63cf27571b91056881b52a6c68/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/8e98f364c292ad7bb2dcbbc3e649b96b25b808f8ead92d8a83f0aa722a26f9d9/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/d4de39928c8863fc25ca8288b36aca3976b19d357e14561de1beb334b7667091/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a95445d5fdefde4231f19b5a176f4d5c7b9051269c909be4ce2a26d5e6b2de5b/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/bd89ecfef5b80ff4a54daa892dd3c37b141684ea9976aa3763ad3e7a29d82bf2/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/95df617aee3eb50df8eb20cb3ed039b970c159109a1ab4b6baf07188bab926d1/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/5ed938d15dee975da759cf9c70da12ca069acb8071502c20ef9da79896887519/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/15bb3390fa54f6cdcc4c2ab29b4a10c367f10cc7dcff826da2fb146d234086e9/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/db28e5d1ae09bc9a515ec2c79a0ad085bb151be6ba3ab4ed76e4a205792e4e44/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2cab715202e2586bbb7e51ace7520b116f0014b322395471bb9558ff7cf8fdfa/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7d75a755b97f6167f3bf9aa5d1160388141364f71872f4db010d2f4b2b159acc/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7bb7c95e0daca7581f6dd7e0bfcdc510ffa8f03d1345e7d704b386d94229b240/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7febce5e03915bffda0ff810a979be6e260b89724afe4486fb6f3a2eb78ce3c7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/d7328b7cd4a48d9c43faef5ae6edfff9f5ef12c2f216692b44af745f4d17aa93/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5429ac7ee3179c6d91c6b5d81c5d0ddce14e4b720f84d85517663d8c7792b53d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a46e513d09df9e136927bdda80de79609476113fadbdf8f657d313fc1fb60105/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.434687 5272 manager.go:217] Machine: {Timestamp:2026-11-03 22:23:39.434234406 +0000 UTC m=+0.056542705 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100041 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c14ce8157ce113a9d2443a493ce00 SystemUUID:ec2c14ce-8157-ce11-3a9d-2443a493ce00 BootID:f6090455-3819-455a-bb6a-a637e53451d5 Filesystems:[{Device:/var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volumes/kubernetes.io~projected/kube-api-access-n7q4p DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40f83a693c2d0ba8e95542b742cd35bf99e3680d7be70679d5003674059e4bdc/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6944e293409bbb472dbc1bf7fa7777f7664b4d280daedc3535aa061e7f5e6889/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:281 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48fecb37-0646-4895-9958-ac184f9356bd/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:332 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e825eb762da5ec40f3e662bc6c89e25fda32fc5733b1bc4898f3412f0ee1399/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~projected/kube-api-access-jh4vb DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48fecb37-0646-4895-9958-ac184f9356bd/volumes/kubernetes.io~projected/kube-api-access-fv8z7 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9443ca41-9780-4ecc-94c4-a1d99f138a59/volumes/kubernetes.io~projected/kube-api-access-j2dp7 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de70e8cd-d2c4-45b4-bcd4-9a726497f781/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/908bcf40-a8a2-4655-8561-473df410dfb0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c213277-1f38-4e9b-8e2f-50c93be267e0/volumes/kubernetes.io~projected/kube-api-access-schhd DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6adef5f-a8df-4321-a2c3-ea90ca555c52/volumes/kubernetes.io~projected/kube-api-access-2r484 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b9b949c-4919-4d59-ae4d-e0083278cfed/volumes/kubernetes.io~projected/kube-api-access-kcpg9 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ac28a61-63eb-4ac5-a145-73045bf8111d/volumes/kubernetes.io~projected/kube-api-access-7w6sz DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb6002e7cb3e22de0b69530b64f8473e24d0b83e774b6da04ad653b6b0fef46a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aef90eda4fe9cb75d5fc2cd2d9bd677ca110c8f9da700ab252843435cea2ab91/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1dcebfc-8c28-444b-92de-5459830b78f6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af46fe6c-4921-4ca7-bb85-eec4829769af/volumes/kubernetes.io~projected/kube-api-access-rd59w DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab587ad5be1dbec779f642ff6a96be22cc156b4ad24b1fa917e54f63c50ec3ce/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41972913-8aa1-4d44-97b0-033ba10133f5/volumes/kubernetes.io~projected/kube-api-access-4jrqs DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f337dbbdbaa4621ab587f3d59dae1c799f43000e4f45d32c8e18e8131dc71ba8/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19283361fd7e14c1606d2adcea32f2b5be11ecd0733aaab3bda5bff9eef9955e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd763789-7d6e-4c46-b360-05c03f3d112a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bf11df9abb1f56b533b89d2177d9a5638b6ac7efa801e0f28bfaef44cb6fbf0/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57cc701c9438a0dcda7e3588f029ae67bb7358e3180f37f49fc8a581ab23aec5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ff8207d7c92cf76de93a5e2d9530881b1677c182839c61014f535a2538cca79/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a2881635a11f5415e17d9be84d351cc23675fd5cc413671a7c4997502fe26c9/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3/volumes/kubernetes.io~projected/kube-api-access-xz9tx DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e8f6897c1cbf9a54c059bcc859fd4ab5eeb4a0edaf13305062d06d5a4e8df22/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~projected/kube-api-access-4s6gn DeviceMajor:0 DeviceMinor:255 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2e63ee2-e60b-45fd-90e4-e60785022ff7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1dcebfc-8c28-444b-92de-5459830b78f6/volumes/kubernetes.io~projected/kube-api-access-mn7p8 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92cf61d0be80e7241a3bfa3a96080d37536c00b6f27f522b0025a9e92278f7e9/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbc3e2c813389af4d0382b2f9b3e4debfd66d748acfa84716b73f659599764f5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/838971d1-4352-4fa8-80ec-f45ceb86b53c/volumes/kubernetes.io~projected/kube-api-access-j74p2 DeviceMajor:0 DeviceMinor:254 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01c3b879-44a4-435d-ae16-acc5c57992b0/volumes/kubernetes.io~projected/kube-api-access-66hzt DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2636ee5d8aeaa810525fd6a4f466d8279f60feed51558d9ae2780aad25e0ef9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c213277-1f38-4e9b-8e2f-50c93be267e0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd763789-7d6e-4c46-b360-05c03f3d112a/volumes/kubernetes.io~projected/kube-api-access-c84gm DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ac641f1-837b-47f2-8aaf-09d93df2be1b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a2881635a11f54 MacAddress:fe:55:13:9a:f4:1a Speed:10000 Mtu:8901} {Name:40f83a693c2d0ba MacAddress:fa:62:95:1e:d6:d5 Speed:10000 Mtu:8901} {Name:4ff8207d7c92cf7 MacAddress:de:63:be:29:02:08 Speed:10000 Mtu:8901} {Name:5bf11df9abb1f56 MacAddress:8a:89:bb:78:ab:18 Speed:10000 Mtu:8901} {Name:6944e293409bbb4 MacAddress:82:f4:d0:30:66:7d Speed:10000 Mtu:8901} {Name:92cf61d0be80e72 MacAddress:52:a1:19:dc:68:bb Speed:10000 Mtu:8901} {Name:9795ceb307f9a89 MacAddress:aa:5d:96:af:b2:58 Speed:10000 Mtu:8901} {Name:ab587ad5be1dbec MacAddress:8e:b4:81:0b:60:d4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f6:56:12:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:6b:9d:12:be:bb Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:f6:56:12:33 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f6:a0:a8:38:88:ab Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:7d:69:d4:da:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435030 5272 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435075 5272 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435331 5272 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435352 5272 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-222.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435479 5272 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435487 5272 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435498 5272 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435516 5272 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435590 5272 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435731 5272 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435860 5272 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435899 5272 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435912 5272 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435921 5272 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435931 5272 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.435939 5272 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.437486 5272 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.437503 5272 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:23:39.440109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.438595 5272 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:23:39.441922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.441907 5272 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:23:39.441922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.441922 5272 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:23:39.442025 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442015 5272 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:23:39.442212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442201 5272 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:23:39.442242 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442230 5272 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:23:39.442330 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442320 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:23:39.442375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442335 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:23:39.442375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442342 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:23:39.442375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442348 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:23:39.442375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442354 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:23:39.442375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442360 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:23:39.442375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442367 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:23:39.442375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442376 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:23:39.442583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442384 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:23:39.442583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442389 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:23:39.442583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442398 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:23:39.442583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442408 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:23:39.442583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442432 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:23:39.442583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.442440 5272 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:23:39.445778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.445765 5272 server.go:1263] "Started kubelet" Nov 03 22:23:39.445896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.445867 5272 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:23:39.445960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.445873 5272 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:23:39.446012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.445959 5272 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:23:39.446337 ip-10-0-128-222 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:23:39.446475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.446337 5272 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:23:39.447174 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.447155 5272 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:23:39.451447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.451426 5272 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:23:39.452108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.452080 5272 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:23:39.452223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.452161 5272 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:23:39.452223 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.452165 5272 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" Nov 03 22:23:39.452223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.452177 5272 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:23:39.452223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.452190 5272 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:23:39.452418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.452220 5272 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:17:29 +0000 UTC" deadline="2028-06-05 11:58:35.222749447 +0000 UTC" Nov 03 22:23:39.452418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.452245 5272 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13909h34m55.770506221s" Nov 03 22:23:39.453071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.453049 5272 factory.go:55] Registering systemd factory Nov 03 22:23:39.453154 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.453075 5272 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:23:39.453604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.453584 5272 factory.go:153] Registering CRI-O factory Nov 03 22:23:39.453692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.453609 5272 factory.go:223] Registration of the crio container factory successfully Nov 03 22:23:39.453692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.453658 5272 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:23:39.453692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.453678 5272 factory.go:103] Registering Raw factory Nov 03 22:23:39.453863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.453694 5272 manager.go:1199] Started watching for new ooms in manager Nov 03 22:23:39.454059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454044 5272 manager.go:319] Starting recovery of all containers Nov 03 22:23:39.454608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454576 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908bcf40-a8a2-4655-8561-473df410dfb0" volumeName="kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-config" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454607 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2e63ee2-e60b-45fd-90e4-e60785022ff7" volumeName="kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454619 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454631 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd763789-7d6e-4c46-b360-05c03f3d112a" volumeName="kubernetes.io/secret/dd763789-7d6e-4c46-b360-05c03f3d112a-cert" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454639 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac641f1-837b-47f2-8aaf-09d93df2be1b" volumeName="kubernetes.io/projected/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-api-access-4s6gn" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454647 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="838971d1-4352-4fa8-80ec-f45ceb86b53c" volumeName="kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-metrics-client-ca" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454658 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="838971d1-4352-4fa8-80ec-f45ceb86b53c" volumeName="kubernetes.io/empty-dir/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-textfile" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454666 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db" seLinuxMountContext="" Nov 03 22:23:39.454679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454679 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac28a61-63eb-4ac5-a145-73045bf8111d" volumeName="kubernetes.io/projected/1ac28a61-63eb-4ac5-a145-73045bf8111d-kube-api-access-7w6sz" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454688 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9443ca41-9780-4ecc-94c4-a1d99f138a59" volumeName="kubernetes.io/projected/9443ca41-9780-4ecc-94c4-a1d99f138a59-kube-api-access-j2dp7" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454698 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6adef5f-a8df-4321-a2c3-ea90ca555c52" volumeName="kubernetes.io/projected/b6adef5f-a8df-4321-a2c3-ea90ca555c52-kube-api-access-2r484" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454707 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454715 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8edba00c-b9a2-4437-a117-4cbd8c7f3fc3" volumeName="kubernetes.io/configmap/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454725 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908bcf40-a8a2-4655-8561-473df410dfb0" volumeName="kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-env-overrides" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454752 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454761 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01c3b879-44a4-435d-ae16-acc5c57992b0" volumeName="kubernetes.io/projected/01c3b879-44a4-435d-ae16-acc5c57992b0-kube-api-access-66hzt" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454769 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac641f1-837b-47f2-8aaf-09d93df2be1b" volumeName="kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454780 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="838971d1-4352-4fa8-80ec-f45ceb86b53c" volumeName="kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454788 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac641f1-837b-47f2-8aaf-09d93df2be1b" volumeName="kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-metrics-client-ca" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454798 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454824 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1dcebfc-8c28-444b-92de-5459830b78f6" volumeName="kubernetes.io/empty-dir/f1dcebfc-8c28-444b-92de-5459830b78f6-tmp-dir" seLinuxMountContext="" Nov 03 22:23:39.455042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.454839 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac28a61-63eb-4ac5-a145-73045bf8111d" volumeName="kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:23:39.455608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.455077 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac28a61-63eb-4ac5-a145-73045bf8111d" volumeName="kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-binary-copy" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.455096 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac641f1-837b-47f2-8aaf-09d93df2be1b" volumeName="kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456367 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c213277-1f38-4e9b-8e2f-50c93be267e0" volumeName="kubernetes.io/empty-dir/7c213277-1f38-4e9b-8e2f-50c93be267e0-data-volume" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456385 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41972913-8aa1-4d44-97b0-033ba10133f5" volumeName="kubernetes.io/empty-dir/41972913-8aa1-4d44-97b0-033ba10133f5-tmp-dir" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456401 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48fecb37-0646-4895-9958-ac184f9356bd" volumeName="kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456415 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908bcf40-a8a2-4655-8561-473df410dfb0" volumeName="kubernetes.io/secret/908bcf40-a8a2-4655-8561-473df410dfb0-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456436 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456448 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456463 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="838971d1-4352-4fa8-80ec-f45ceb86b53c" volumeName="kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:23:39.456478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456477 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456492 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456505 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de70e8cd-d2c4-45b4-bcd4-9a726497f781" volumeName="kubernetes.io/secret/de70e8cd-d2c4-45b4-bcd4-9a726497f781-agent-certs" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456517 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41972913-8aa1-4d44-97b0-033ba10133f5" volumeName="kubernetes.io/projected/41972913-8aa1-4d44-97b0-033ba10133f5-kube-api-access-4jrqs" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456532 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c213277-1f38-4e9b-8e2f-50c93be267e0" volumeName="kubernetes.io/configmap/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456545 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="838971d1-4352-4fa8-80ec-f45ceb86b53c" volumeName="kubernetes.io/projected/838971d1-4352-4fa8-80ec-f45ceb86b53c-kube-api-access-j74p2" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456559 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9443ca41-9780-4ecc-94c4-a1d99f138a59" volumeName="kubernetes.io/configmap/9443ca41-9780-4ecc-94c4-a1d99f138a59-serviceca" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456573 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6adef5f-a8df-4321-a2c3-ea90ca555c52" volumeName="kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cni-binary-copy" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456589 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456600 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456612 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1dcebfc-8c28-444b-92de-5459830b78f6" volumeName="kubernetes.io/secret/f1dcebfc-8c28-444b-92de-5459830b78f6-metrics-tls" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456624 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac641f1-837b-47f2-8aaf-09d93df2be1b" volumeName="kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456635 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c213277-1f38-4e9b-8e2f-50c93be267e0" volumeName="kubernetes.io/projected/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-api-access-schhd" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456647 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac28a61-63eb-4ac5-a145-73045bf8111d" volumeName="kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456658 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8edba00c-b9a2-4437-a117-4cbd8c7f3fc3" volumeName="kubernetes.io/projected/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-kube-api-access-xz9tx" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456670 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908bcf40-a8a2-4655-8561-473df410dfb0" volumeName="kubernetes.io/projected/908bcf40-a8a2-4655-8561-473df410dfb0-kube-api-access-n7q4p" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456682 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1dcebfc-8c28-444b-92de-5459830b78f6" volumeName="kubernetes.io/projected/f1dcebfc-8c28-444b-92de-5459830b78f6-kube-api-access-mn7p8" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456693 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac641f1-837b-47f2-8aaf-09d93df2be1b" volumeName="kubernetes.io/empty-dir/2ac641f1-837b-47f2-8aaf-09d93df2be1b-volume-directive-shadow" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456736 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="838971d1-4352-4fa8-80ec-f45ceb86b53c" volumeName="kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-tls" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456748 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af46fe6c-4921-4ca7-bb85-eec4829769af" volumeName="kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456761 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b9b949c-4919-4d59-ae4d-e0083278cfed" volumeName="kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-tmp" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456772 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908bcf40-a8a2-4655-8561-473df410dfb0" volumeName="kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456825 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456838 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd763789-7d6e-4c46-b360-05c03f3d112a" volumeName="kubernetes.io/projected/dd763789-7d6e-4c46-b360-05c03f3d112a-kube-api-access-c84gm" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456849 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6adef5f-a8df-4321-a2c3-ea90ca555c52" volumeName="kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-daemon-config" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456871 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48fecb37-0646-4895-9958-ac184f9356bd" volumeName="kubernetes.io/projected/48fecb37-0646-4895-9958-ac184f9356bd-kube-api-access-fv8z7" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456886 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b9b949c-4919-4d59-ae4d-e0083278cfed" volumeName="kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-tuned" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456899 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" volumeName="kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456914 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de70e8cd-d2c4-45b4-bcd4-9a726497f781" volumeName="kubernetes.io/configmap/de70e8cd-d2c4-45b4-bcd4-9a726497f781-konnectivity-ca" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456927 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1dcebfc-8c28-444b-92de-5459830b78f6" volumeName="kubernetes.io/configmap/f1dcebfc-8c28-444b-92de-5459830b78f6-config-volume" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456939 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b9b949c-4919-4d59-ae4d-e0083278cfed" volumeName="kubernetes.io/projected/6b9b949c-4919-4d59-ae4d-e0083278cfed-kube-api-access-kcpg9" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456955 5272 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c213277-1f38-4e9b-8e2f-50c93be267e0" volumeName="kubernetes.io/secret/7c213277-1f38-4e9b-8e2f-50c93be267e0-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456968 5272 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:23:39.456954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.456976 5272 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:23:39.483737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.483704 5272 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:23:39.485046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.485019 5272 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:23:39.485046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.485049 5272 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:23:39.485239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.485069 5272 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:23:39.485239 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.485102 5272 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:23:39.490494 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.490465 5272 generic.go:356] "Generic (PLEG): container finished" podID="838971d1-4352-4fa8-80ec-f45ceb86b53c" containerID="82c08db1d9799a9559831b1ba23725a4c2fcd409f40464680758727d0287b7ba" exitCode=0 Nov 03 22:23:39.525983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.525950 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="7d052b166edaa4212936fbeb0fba68abb03f4de8d7b0d9dcc1766f272154be4a" exitCode=0 Nov 03 22:23:39.525983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.525977 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="b4c2b9c295ee5ada60dd0f092fad69eee7373acfc7a56fd8eb85b8b5effbf8ab" exitCode=0 Nov 03 22:23:39.525983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.525986 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="f906e0bf2c6ae84646d81d78811986d7b3c9357976dd118df30c0c9044e83913" exitCode=0 Nov 03 22:23:39.526274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.525996 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="66da736b12e23f123b4083d505d11106f2d912bc60134cd8cd0756097437ff48" exitCode=0 Nov 03 22:23:39.526274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.526003 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="02539120311a1634a97a2179d095db23a502e187b91dd61a7813ba853a31177f" exitCode=0 Nov 03 22:23:39.526274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.526008 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ac28a61-63eb-4ac5-a145-73045bf8111d" containerID="59e1bf32fc6e38526babda8d024e5f2c1bbed9dc8ce89c2818f1c59ba91d55db" exitCode=0 Nov 03 22:23:39.529255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.529229 5272 generic.go:356] "Generic (PLEG): container finished" podID="5aaf8075c83069da4e0349ec07734525" containerID="82eca44b8f7b3c182b0e2c879baa7f4852dbfa269abc452736d616bf585551de" exitCode=0 Nov 03 22:23:39.534667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.534641 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7" exitCode=0 Nov 03 22:23:39.546306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.546283 5272 manager.go:324] Recovery completed Nov 03 22:23:39.552229 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.552207 5272 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" Nov 03 22:23:39.563906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.563888 5272 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:23:39.566297 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.566281 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:23:39.566378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.566312 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:23:39.566378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.566328 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:23:39.568930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.568912 5272 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:23:39.568930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.568925 5272 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:23:39.569020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.568926 5272 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:23:39.569060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569025 5272 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:23:39.569193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569174 5272 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:23:39.569240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569194 5272 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:23:39.569240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569223 5272 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:23:39.569240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569232 5272 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:23:39.569240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569240 5272 policy_none.go:50] "Start" Nov 03 22:23:39.569373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569253 5272 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:23:39.569373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569266 5272 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:23:39.569772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569756 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:23:39.569906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569788 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:23:39.569906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.569822 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:23:39.571989 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.571974 5272 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:23:39.572030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.571992 5272 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:23:39.572030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.571999 5272 policy_none.go:44] "Start" Nov 03 22:23:39.576171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.576113 5272 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:23:39.578745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.578733 5272 manager.go:341] "Starting Device Plugin manager" Nov 03 22:23:39.578793 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.578757 5272 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:23:39.578793 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.578767 5272 server.go:87] "Starting device plugin registration server" Nov 03 22:23:39.579104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.579089 5272 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:23:39.579167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.579105 5272 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:23:39.579221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.579172 5272 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:23:39.579291 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.579260 5272 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:23:39.579346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.579299 5272 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:23:39.579346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.579306 5272 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:23:39.579512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.579426 5272 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:23:39.579288772Z","UUID":"641d8800-6891-4863-b6c2-9771a4ada3b2","Handler":null,"Name":"","Endpoint":""} Nov 03 22:23:39.580936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.580916 5272 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:23:39.581016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.580958 5272 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:23:39.581123 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.581100 5272 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:23:39.581199 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.581151 5272 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" Nov 03 22:23:39.585407 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.585367 5272 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal"] Nov 03 22:23:39.585520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.585508 5272 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:23:39.586574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.586555 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:23:39.586677 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.586635 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:23:39.586677 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.586654 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:23:39.586955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.586943 5272 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:23:39.587105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.587089 5272 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:23:39.587698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.587682 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:23:39.587767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.587706 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:23:39.587767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.587716 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:23:39.587767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.587740 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:23:39.587767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.587762 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:23:39.587935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.587772 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:23:39.587935 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.587850 5272 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.588059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.588031 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"d79e0f53b5b82fd1a135f4636735856a","Type":"ContainerStarted","Data":"cce60f8e5fdd9b268b6a889325976f6e88363943c4e4a95a8bec19fa56234692"} Nov 03 22:23:39.588098 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.588067 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"d79e0f53b5b82fd1a135f4636735856a","Type":"ContainerStarted","Data":"e4a8d1178be390c562a1fadcd42ec042d7e200bd4a57d53a631bed72f5ddba23"} Nov 03 22:23:39.588131 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.588121 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"5aaf8075c83069da4e0349ec07734525","Type":"ContainerStarted","Data":"9552617530a51bf16d2ec4ac3558837c036cea201a0770dbf8375c4079493afb"} Nov 03 22:23:39.588170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.588138 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"5aaf8075c83069da4e0349ec07734525","Type":"ContainerDied","Data":"82eca44b8f7b3c182b0e2c879baa7f4852dbfa269abc452736d616bf585551de"} Nov 03 22:23:39.588170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.588145 5272 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:23:39.588170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.588151 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" event={"ID":"5aaf8075c83069da4e0349ec07734525","Type":"ContainerStarted","Data":"80d2c28f030737bde7d042480c308346c21e48b4684e90d3f2eded0063900a22"} Nov 03 22:23:39.588994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.588976 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:23:39.589036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.589000 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:23:39.589036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.589010 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:23:39.589150 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.589127 5272 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-222.us-west-2.compute.internal\" not found" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.643127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.643101 5272 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:23:39.643218 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.643156 5272 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:23:39.680224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.680197 5272 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:23:39.681192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.681176 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:23:39.681256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.681207 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:23:39.681256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.681220 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:23:39.681256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.681253 5272 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.689911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.689878 5272 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:23:39.735613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.735592 5272 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:23:39.736161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.736144 5272 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:23:39.743382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.743365 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.743466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.743397 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.743466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.743426 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d79e0f53b5b82fd1a135f4636735856a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"d79e0f53b5b82fd1a135f4636735856a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.753573 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.753552 5272 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.788011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.787988 5272 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.788070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.788043 5272 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.788590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.788576 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeNotReady" Nov 03 22:23:39.788644 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.788603 5272 setters.go:546] "Node became not ready" node="ip-10-0-128-222.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T22:23:39Z","lastTransitionTime":"2026-11-03T22:23:39Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 22:23:39.811119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.811102 5272 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:23:39.811171 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.811140 5272 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.811171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.811153 5272 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.843712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.843650 5272 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-222.us-west-2.compute.internal" is owned by different node (oldNodeID="d60a8fb3-9247-489a-be17-13cb41006121", newNodeID=""), cleaning up... Nov 03 22:23:39.843712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.843676 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.843712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.843701 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.843917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.843719 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d79e0f53b5b82fd1a135f4636735856a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"d79e0f53b5b82fd1a135f4636735856a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.843917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.843762 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d79e0f53b5b82fd1a135f4636735856a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"d79e0f53b5b82fd1a135f4636735856a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.843917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.843785 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.843917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.843791 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5aaf8075c83069da4e0349ec07734525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" (UID: \"5aaf8075c83069da4e0349ec07734525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:39.869445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:39.869424 5272 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:23:39.869537 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:39.869467 5272 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:40.436857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.436830 5272 apiserver.go:52] "Watching apiserver" Nov 03 22:23:40.536860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.536815 5272 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:23:40.537624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.537593 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xfjg6","openshift-ingress-canary/ingress-canary-h6v68","openshift-cluster-node-tuning-operator/tuned-g2gcl","openshift-multus/multus-6rv4x","kube-system/konnectivity-agent-lrdk5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5","openshift-image-registry/node-ca-95xrs","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-gcrp2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-c54qw","kube-system/global-pull-secret-syncer-zlkqx","openshift-network-diagnostics/network-check-target-hdq68","openshift-ovn-kubernetes/ovnkube-node-fq4b6","openshift-insights/insights-runtime-extractor-54cfh","kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-x22cf","openshift-dns/node-resolver-bhsqf","openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb","openshift-dns/dns-default-72kwh"] Nov 03 22:23:40.538208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.538182 5272 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:40.538312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.538263 5272 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:40.553756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.553735 5272 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:23:40.588440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.588411 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kccnl\"" Nov 03 22:23:40.588543 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.588527 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.588611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.588598 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.588669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.588656 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:23:40.588914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.588893 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.588977 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.588911 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mdpv5\"" Nov 03 22:23:40.588977 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.588941 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.589106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.589090 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:23:40.589159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.589148 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:23:40.592493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.592477 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:23:40.595924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.595905 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.595924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.595918 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:23:40.596050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.595905 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.596050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.595916 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:23:40.596050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.595956 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:23:40.596050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.595961 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:23:40.596050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.595987 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ztkgs\"" Nov 03 22:23:40.596050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596019 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.596314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596086 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:23:40.596314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596101 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.596314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596107 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hvj2k\"" Nov 03 22:23:40.596314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596150 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.596314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596171 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ljhlr\"" Nov 03 22:23:40.596314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596194 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.596314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596269 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hp7n8\"" Nov 03 22:23:40.596510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.596372 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:23:40.598527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.598512 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kfchj\"" Nov 03 22:23:40.598579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.598539 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.598579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.598548 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.598653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.598621 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.598688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.598652 5272 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:23:40.598718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.598686 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nh9f8\"" Nov 03 22:23:40.598718 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:40.598686 5272 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:40.598824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.598652 5272 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:23:40.598873 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:40.598857 5272 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" Nov 03 22:23:40.647934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.647910 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.647934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.647937 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84gm\" (UniqueName: \"kubernetes.io/projected/dd763789-7d6e-4c46-b360-05c03f3d112a-kube-api-access-c84gm\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:40.648042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.647955 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-run\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.647970 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-dbus\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.648042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.647996 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.648134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648053 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-metrics-client-ca\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.648134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648104 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-bin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.648194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648141 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-modprobe-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648160 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:40.648253 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648224 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-root\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.648282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648250 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-tls\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.648338 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648314 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-socket-dir-parent\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.648384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648369 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-conf-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.648441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648420 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.648482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648471 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-kubernetes\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648503 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-sys\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648534 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-tmp\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648552 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-etc-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.648612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648581 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-config\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.648640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648610 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-conf\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648630 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-kubelet-config\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.648693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648653 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.648721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648698 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-tmp\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648816 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-node-log\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.648935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648852 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-registration-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.648935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648860 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-config\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.648935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648869 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9443ca41-9780-4ecc-94c4-a1d99f138a59-serviceca\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:23:40.648935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648886 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-kubelet\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.648935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648912 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-tuned\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.648935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648934 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-netd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.649117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.648980 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de70e8cd-d2c4-45b4-bcd4-9a726497f781-agent-certs\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:23:40.649117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649003 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-tuned\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.649117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649017 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-system-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.649198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649142 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.649198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649178 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6sz\" (UniqueName: \"kubernetes.io/projected/1ac28a61-63eb-4ac5-a145-73045bf8111d-kube-api-access-7w6sz\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.649291 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649275 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.649321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649288 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de70e8cd-d2c4-45b4-bcd4-9a726497f781-agent-certs\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:23:40.649321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649307 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-host\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.649377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649347 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-k8s-cni-cncf-io\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.649409 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649384 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-kubelet\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.649442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649419 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-slash\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.649475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649455 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd763789-7d6e-4c46-b360-05c03f3d112a-cert\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:40.649475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649472 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.649537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649487 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-systemd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.649537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649503 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-script-lib\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.649593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649539 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41972913-8aa1-4d44-97b0-033ba10133f5-tmp-dir\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:23:40.649593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649565 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv8z7\" (UniqueName: \"kubernetes.io/projected/48fecb37-0646-4895-9958-ac184f9356bd-kube-api-access-fv8z7\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:40.649593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649579 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.649593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649585 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-sys-fs\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.649745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649604 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41972913-8aa1-4d44-97b0-033ba10133f5-tmp-dir\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:23:40.649745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649614 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-textfile\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.649745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649737 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.649882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649769 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-textfile\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.649882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649826 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-bin\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.649882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649839 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-ovnkube-script-lib\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.649882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649847 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysconfig\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.649882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649868 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.649882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649869 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649935 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649961 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-device-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.649980 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650002 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c213277-1f38-4e9b-8e2f-50c93be267e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650019 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650034 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41972913-8aa1-4d44-97b0-033ba10133f5-hosts-file\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650049 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-iptables-alerter-script\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:23:40.650079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650069 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cni-binary-copy\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.650366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650108 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.650366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650143 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-var-lib-kubelet\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.650366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650184 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.650366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650239 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de70e8cd-d2c4-45b4-bcd4-9a726497f781-konnectivity-ca\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:23:40.650366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650312 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-iptables-alerter-script\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:23:40.650366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650351 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-log-socket\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.650545 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650400 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpg9\" (UniqueName: \"kubernetes.io/projected/6b9b949c-4919-4d59-ae4d-e0083278cfed-kube-api-access-kcpg9\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.650666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650652 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cni-binary-copy\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.650731 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650682 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de70e8cd-d2c4-45b4-bcd4-9a726497f781-konnectivity-ca\") pod \"konnectivity-agent-lrdk5\" (UID: \"de70e8cd-d2c4-45b4-bcd4-9a726497f781\") " pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:23:40.650779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650738 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.650921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650902 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-etc-kubernetes\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.650958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650941 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r484\" (UniqueName: \"kubernetes.io/projected/b6adef5f-a8df-4321-a2c3-ea90ca555c52-kube-api-access-2r484\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.650998 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650959 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9tx\" (UniqueName: \"kubernetes.io/projected/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-kube-api-access-xz9tx\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:23:40.650998 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.650977 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908bcf40-a8a2-4655-8561-473df410dfb0-ovn-node-metrics-cert\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.651086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651075 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9443ca41-9780-4ecc-94c4-a1d99f138a59-host\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:23:40.651154 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651103 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cnibin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.651154 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651147 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/908bcf40-a8a2-4655-8561-473df410dfb0-ovn-node-metrics-cert\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.651230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651152 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4vb\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.651230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651173 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-var-lib-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.651230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651200 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ac641f1-837b-47f2-8aaf-09d93df2be1b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.651333 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651263 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ac641f1-837b-47f2-8aaf-09d93df2be1b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.651333 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651297 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-systemd\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.651411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651343 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.651411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651359 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dcebfc-8c28-444b-92de-5459830b78f6-tmp-dir\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.651411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651375 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.651529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651416 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.651529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651435 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-hostroot\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.651529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651448 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dcebfc-8c28-444b-92de-5459830b78f6-tmp-dir\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.651529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651479 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.651529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651498 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7p8\" (UniqueName: \"kubernetes.io/projected/f1dcebfc-8c28-444b-92de-5459830b78f6-kube-api-access-mn7p8\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.651529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651516 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6gn\" (UniqueName: \"kubernetes.io/projected/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-api-access-4s6gn\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651545 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651601 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74p2\" (UniqueName: \"kubernetes.io/projected/838971d1-4352-4fa8-80ec-f45ceb86b53c-kube-api-access-j74p2\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651631 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-ovn\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651658 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651696 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hzt\" (UniqueName: \"kubernetes.io/projected/01c3b879-44a4-435d-ae16-acc5c57992b0-kube-api-access-66hzt\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651726 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c213277-1f38-4e9b-8e2f-50c93be267e0-data-volume\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651760 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-lib-modules\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651775 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-wtmp\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651792 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.651820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651822 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c213277-1f38-4e9b-8e2f-50c93be267e0-data-volume\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651837 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-multus\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651866 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.651881 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652044 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-system-cni-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652066 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-cnibin\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652102 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-env-overrides\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652121 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-sys\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652137 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-multus-certs\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652152 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dcebfc-8c28-444b-92de-5459830b78f6-metrics-tls\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.652260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652167 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652277 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrqs\" (UniqueName: \"kubernetes.io/projected/41972913-8aa1-4d44-97b0-033ba10133f5-kube-api-access-4jrqs\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652306 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652340 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcebfc-8c28-444b-92de-5459830b78f6-config-volume\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652367 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-netns\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652391 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c213277-1f38-4e9b-8e2f-50c93be267e0-crio-socket\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652437 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652450 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/908bcf40-a8a2-4655-8561-473df410dfb0-env-overrides\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652465 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652549 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652571 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652588 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-netns\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652604 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-daemon-config\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.652652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652630 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schhd\" (UniqueName: \"kubernetes.io/projected/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-api-access-schhd\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652689 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652706 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-host-slash\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652723 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7q4p\" (UniqueName: \"kubernetes.io/projected/908bcf40-a8a2-4655-8561-473df410dfb0-kube-api-access-n7q4p\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652737 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652752 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652787 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dp7\" (UniqueName: \"kubernetes.io/projected/9443ca41-9780-4ecc-94c4-a1d99f138a59-kube-api-access-j2dp7\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652826 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-daemon-config\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652841 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-systemd-units\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.653211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.652861 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-socket-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.688534 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.688478 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.688701 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.688684 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:23:40.688747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.688709 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.689010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.688994 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 22:23:40.689083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689066 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 22:23:40.689181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689068 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.689613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689592 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:23:40.689613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689606 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:23:40.689773 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689607 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.689839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689822 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 22:23:40.689839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689824 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 22:23:40.689943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689856 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 22:23:40.689943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689862 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 22:23:40.689943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689900 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g6sdx\"" Nov 03 22:23:40.689943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689905 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:23:40.689943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689905 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.690178 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689976 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:23:40.690178 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.689863 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 22:23:40.690178 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690094 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.690178 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690144 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690185 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690189 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690216 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690223 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690242 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690251 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690261 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:23:40.690357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690305 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mf8dj\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690384 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk9bw\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690441 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2dn77\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690448 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kbfvs\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690503 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c213277-1f38-4e9b-8e2f-50c93be267e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690549 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690553 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690572 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690590 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2kdj\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690613 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690622 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690640 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690663 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2jqrs\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690674 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690665 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v9lfz\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690682 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 22:23:40.690718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690710 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vkjzf\"" Nov 03 22:23:40.691222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690858 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:23:40.691222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690860 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f4nwd\"" Nov 03 22:23:40.691222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.690936 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 22:23:40.692148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692133 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.692525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692497 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.692632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692507 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.692632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692550 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.692743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692652 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.692743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692659 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dcebfc-8c28-444b-92de-5459830b78f6-metrics-tls\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.692743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692682 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fecb37-0646-4895-9958-ac184f9356bd-metrics-certs\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:40.692743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692688 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.692954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692815 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.692954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692820 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcebfc-8c28-444b-92de-5459830b78f6-config-volume\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.692954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692862 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.692954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692910 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.692954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692921 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.693198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.692966 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.693198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.693099 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.698646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.698623 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ac28a61-63eb-4ac5-a145-73045bf8111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.698646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.698633 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-metrics-client-ca\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.698811 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.698650 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-tls\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.698811 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.698738 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e63ee2-e60b-45fd-90e4-e60785022ff7-original-pull-secret\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.698948 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.698914 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.700082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.700061 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd763789-7d6e-4c46-b360-05c03f3d112a-cert\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:40.700156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.700061 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.700247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.700224 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac641f1-837b-47f2-8aaf-09d93df2be1b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.700422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.700399 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.700492 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.700404 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.700540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.700523 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9443ca41-9780-4ecc-94c4-a1d99f138a59-serviceca\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:23:40.736235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736206 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv8z7\" (UniqueName: \"kubernetes.io/projected/48fecb37-0646-4895-9958-ac184f9356bd-kube-api-access-fv8z7\") pod \"network-metrics-daemon-xfjg6\" (UID: \"48fecb37-0646-4895-9958-ac184f9356bd\") " pod="openshift-multus/network-metrics-daemon-xfjg6" Nov 03 22:23:40.736359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736269 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9tx\" (UniqueName: \"kubernetes.io/projected/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-kube-api-access-xz9tx\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:23:40.736359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736272 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6sz\" (UniqueName: \"kubernetes.io/projected/1ac28a61-63eb-4ac5-a145-73045bf8111d-kube-api-access-7w6sz\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.736529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736509 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrqs\" (UniqueName: \"kubernetes.io/projected/41972913-8aa1-4d44-97b0-033ba10133f5-kube-api-access-4jrqs\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:23:40.736529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736518 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74p2\" (UniqueName: \"kubernetes.io/projected/838971d1-4352-4fa8-80ec-f45ceb86b53c-kube-api-access-j74p2\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.736640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736586 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84gm\" (UniqueName: \"kubernetes.io/projected/dd763789-7d6e-4c46-b360-05c03f3d112a-kube-api-access-c84gm\") pod \"ingress-canary-h6v68\" (UID: \"dd763789-7d6e-4c46-b360-05c03f3d112a\") " pod="openshift-ingress-canary/ingress-canary-h6v68" Nov 03 22:23:40.736859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736837 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hzt\" (UniqueName: \"kubernetes.io/projected/01c3b879-44a4-435d-ae16-acc5c57992b0-kube-api-access-66hzt\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.736927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.736839 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpg9\" (UniqueName: \"kubernetes.io/projected/6b9b949c-4919-4d59-ae4d-e0083278cfed-kube-api-access-kcpg9\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.737047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737024 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-schhd\" (UniqueName: \"kubernetes.io/projected/7c213277-1f38-4e9b-8e2f-50c93be267e0-kube-api-access-schhd\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.737129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737029 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7q4p\" (UniqueName: \"kubernetes.io/projected/908bcf40-a8a2-4655-8561-473df410dfb0-kube-api-access-n7q4p\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.737129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737055 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6gn\" (UniqueName: \"kubernetes.io/projected/2ac641f1-837b-47f2-8aaf-09d93df2be1b-kube-api-access-4s6gn\") pod \"kube-state-metrics-5dc8d7d89b-m5xmb\" (UID: \"2ac641f1-837b-47f2-8aaf-09d93df2be1b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-m5xmb" Nov 03 22:23:40.737398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737370 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r484\" (UniqueName: \"kubernetes.io/projected/b6adef5f-a8df-4321-a2c3-ea90ca555c52-kube-api-access-2r484\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.737398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737387 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd59w\" (UniqueName: \"kubernetes.io/projected/af46fe6c-4921-4ca7-bb85-eec4829769af-kube-api-access-rd59w\") pod \"network-check-target-hdq68\" (UID: \"af46fe6c-4921-4ca7-bb85-eec4829769af\") " pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:40.737535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737498 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dp7\" (UniqueName: \"kubernetes.io/projected/9443ca41-9780-4ecc-94c4-a1d99f138a59-kube-api-access-j2dp7\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:23:40.737535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737518 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4vb\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\") pod \"alertmanager-main-0\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:23:40.737601 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.737569 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7p8\" (UniqueName: \"kubernetes.io/projected/f1dcebfc-8c28-444b-92de-5459830b78f6-kube-api-access-mn7p8\") pod \"dns-default-72kwh\" (UID: \"f1dcebfc-8c28-444b-92de-5459830b78f6\") " pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:40.753396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753376 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-hostroot\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753402 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.753485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753419 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-ovn\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.753485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753433 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.753485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753447 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-lib-modules\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.753485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753461 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-wtmp\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.753485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753476 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-multus\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753497 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-system-cni-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753511 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-hostroot\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753514 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753520 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-cnibin\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753561 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-multus\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753514 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-ovn\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753567 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-system-cni-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753574 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753552 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-cnibin\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753582 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-sys\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753598 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-node-exporter-wtmp\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753621 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-sys\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753640 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-lib-modules\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753641 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-multus-certs\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753677 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-netns\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753677 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-multus-certs\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753692 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c213277-1f38-4e9b-8e2f-50c93be267e0-crio-socket\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753708 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753721 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-run-netns\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753734 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-netns\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753754 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c213277-1f38-4e9b-8e2f-50c93be267e0-crio-socket\") pod \"insights-runtime-extractor-54cfh\" (UID: \"7c213277-1f38-4e9b-8e2f-50c93be267e0\") " pod="openshift-insights/insights-runtime-extractor-54cfh" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753760 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-host-slash\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753763 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753788 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-netns\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.753772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753796 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753819 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8edba00c-b9a2-4437-a117-4cbd8c7f3fc3-host-slash\") pod \"iptables-alerter-c54qw\" (UID: \"8edba00c-b9a2-4437-a117-4cbd8c7f3fc3\") " pod="openshift-network-operator/iptables-alerter-c54qw" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753836 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753841 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753861 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-systemd-units\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753875 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-socket-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753894 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-run\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753918 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-dbus\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753920 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-systemd-units\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753942 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-bin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753961 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753963 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-socket-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753966 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-modprobe-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.753993 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-run\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754002 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-cni-bin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754000 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-root\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754027 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838971d1-4352-4fa8-80ec-f45ceb86b53c-root\") pod \"node-exporter-gcrp2\" (UID: \"838971d1-4352-4fa8-80ec-f45ceb86b53c\") " pod="openshift-monitoring/node-exporter-gcrp2" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754013 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-dbus\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754037 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-socket-dir-parent\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754067 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-modprobe-d\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754080 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-conf-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754122 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754124 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-conf-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754145 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-kubernetes\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754075 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-multus-socket-dir-parent\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754168 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-sys\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754193 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-kubernetes\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754192 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-etc-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754202 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ac28a61-63eb-4ac5-a145-73045bf8111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x22cf\" (UID: \"1ac28a61-63eb-4ac5-a145-73045bf8111d\") " pod="openshift-multus/multus-additional-cni-plugins-x22cf" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754223 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-etc-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754215 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-conf\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754251 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-sys\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754255 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-kubelet-config\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754279 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e63ee2-e60b-45fd-90e4-e60785022ff7-kubelet-config\") pod \"global-pull-secret-syncer-zlkqx\" (UID: \"b2e63ee2-e60b-45fd-90e4-e60785022ff7\") " pod="kube-system/global-pull-secret-syncer-zlkqx" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754315 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-node-log\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754343 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-registration-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754366 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-kubelet\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754362 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-node-log\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754393 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-registration-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754405 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-netd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754411 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-var-lib-kubelet\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754376 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysctl-conf\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754431 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-system-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754465 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-netd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754483 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-system-cni-dir\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754491 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-host\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754539 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-host\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754559 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-k8s-cni-cncf-io\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754575 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-kubelet\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754596 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-slash\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754614 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-systemd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754619 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-host-run-k8s-cni-cncf-io\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754633 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-kubelet\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754641 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-sys-fs\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754646 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-slash\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754660 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-run-systemd\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754668 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-bin\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754677 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-sys-fs\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754687 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysconfig\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-device-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754702 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-cni-bin\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754712 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-sysconfig\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754739 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-device-dir\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754767 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754793 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41972913-8aa1-4d44-97b0-033ba10133f5-hosts-file\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754838 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c3b879-44a4-435d-ae16-acc5c57992b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgnq5\" (UID: \"01c3b879-44a4-435d-ae16-acc5c57992b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgnq5" Nov 03 22:23:40.754792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754837 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754854 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41972913-8aa1-4d44-97b0-033ba10133f5-hosts-file\") pod \"node-resolver-bhsqf\" (UID: \"41972913-8aa1-4d44-97b0-033ba10133f5\") " pod="openshift-dns/node-resolver-bhsqf" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754865 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-var-lib-kubelet\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754870 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754893 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-log-socket\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754909 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-etc-kubernetes\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754922 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-var-lib-kubelet\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754931 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9443ca41-9780-4ecc-94c4-a1d99f138a59-host\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754940 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-log-socket\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754946 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cnibin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754953 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-etc-kubernetes\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754971 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9443ca41-9780-4ecc-94c4-a1d99f138a59-host\") pod \"node-ca-95xrs\" (UID: \"9443ca41-9780-4ecc-94c4-a1d99f138a59\") " pod="openshift-image-registry/node-ca-95xrs" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.754982 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-var-lib-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.755006 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6adef5f-a8df-4321-a2c3-ea90ca555c52-cnibin\") pod \"multus-6rv4x\" (UID: \"b6adef5f-a8df-4321-a2c3-ea90ca555c52\") " pod="openshift-multus/multus-6rv4x" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.755010 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-systemd\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.755047 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/908bcf40-a8a2-4655-8561-473df410dfb0-var-lib-openvswitch\") pod \"ovnkube-node-fq4b6\" (UID: \"908bcf40-a8a2-4655-8561-473df410dfb0\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:40.756850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:40.755077 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9b949c-4919-4d59-ae4d-e0083278cfed-etc-systemd\") pod \"tuned-g2gcl\" (UID: \"6b9b949c-4919-4d59-ae4d-e0083278cfed\") " pod="openshift-cluster-node-tuning-operator/tuned-g2gcl" Nov 03 22:23:41.008582 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:41.008506 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lrdk5" Nov 03 22:23:41.398246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:41.398211 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-72kwh" Nov 03 22:23:42.223636 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:42.223610 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:42.243484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:42.243457 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:42.544322 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:42.544295 5272 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:23:46.889885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:46.889848 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:23:46.951789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:46.951763 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq4b6" Nov 03 22:23:50.205134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:23:50.205063 5272 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-222.us-west-2.compute.internal" event="NodeReady" Nov 03 22:23:54.498397 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:23:54.498357 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:24:06.823561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:06.823523 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:24:06.823999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:06.823823 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="alertmanager" containerID="cri-o://632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd" gracePeriod=120 Nov 03 22:24:06.823999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:06.823871 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-metric" containerID="cri-o://f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384" gracePeriod=120 Nov 03 22:24:06.823999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:06.823940 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-web" containerID="cri-o://f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1" gracePeriod=120 Nov 03 22:24:06.823999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:06.823948 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="prom-label-proxy" containerID="cri-o://ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb" gracePeriod=120 Nov 03 22:24:06.824195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:06.824004 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy" containerID="cri-o://decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec" gracePeriod=120 Nov 03 22:24:06.824195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:06.823948 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="config-reloader" containerID="cri-o://2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2" gracePeriod=120 Nov 03 22:24:07.624692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624661 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb" exitCode=0 Nov 03 22:24:07.624692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624685 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec" exitCode=0 Nov 03 22:24:07.624692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624692 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2" exitCode=0 Nov 03 22:24:07.624692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624698 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd" exitCode=0 Nov 03 22:24:07.625081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624726 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb"} Nov 03 22:24:07.625081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624789 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec"} Nov 03 22:24:07.625081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624831 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2"} Nov 03 22:24:07.625081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:07.624856 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd"} Nov 03 22:24:08.064729 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.064704 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.113749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.113722 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.113844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.113759 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.113844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.113786 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.113844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.113823 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.113971 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.113859 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.113971 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.113926 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.114070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.113998 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.114070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114032 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.114177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114066 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.114177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114111 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.114177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114143 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.114177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114171 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.114428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114197 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\") pod \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\" (UID: \"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62\") " Nov 03 22:24:08.115706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114513 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:08.115706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.114610 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:08.115706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.115430 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:08.117699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.117663 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:08.117798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.117761 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "kube-api-access-jh4vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:08.117907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.117822 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:08.118010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.117983 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:08.118168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.118142 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:08.118287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.118269 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:08.118350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.118326 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:08.118478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.118460 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:08.120718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.120653 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:08.126367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.126345 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config" pod "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" (UID: "c0a79337-e1e4-40a7-a8cb-5fc3264f9a62"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:08.215056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.214992 5272 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-cluster-tls-config\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215017 5272 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215028 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh4vb\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-kube-api-access-jh4vb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215038 5272 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215048 5272 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-tls-assets\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215057 5272 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-main-db\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215066 5272 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-web-config\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215074 5272 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215083 5272 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-metrics-client-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215092 5272 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215102 5272 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-out\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215110 5272 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-secret-alertmanager-main-tls\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.215306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.215119 5272 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:08.631508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.631468 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384" exitCode=0 Nov 03 22:24:08.631508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.631501 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerID="f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1" exitCode=0 Nov 03 22:24:08.631752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.631556 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384"} Nov 03 22:24:08.631752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.631584 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.631752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.631651 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1"} Nov 03 22:24:08.631752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.631690 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0a79337-e1e4-40a7-a8cb-5fc3264f9a62","Type":"ContainerDied","Data":"9795ceb307f9a89b5b567cd3eca5cd429a270d153ff737a72e2d8788005ebfb9"} Nov 03 22:24:08.631752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.631720 5272 scope.go:122] "RemoveContainer" containerID="ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb" Nov 03 22:24:08.639322 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.639169 5272 scope.go:122] "RemoveContainer" containerID="f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384" Nov 03 22:24:08.645474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.645458 5272 scope.go:122] "RemoveContainer" containerID="decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec" Nov 03 22:24:08.651592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.651570 5272 scope.go:122] "RemoveContainer" containerID="f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1" Nov 03 22:24:08.657541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.657523 5272 scope.go:122] "RemoveContainer" containerID="2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2" Nov 03 22:24:08.664733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.664710 5272 scope.go:122] "RemoveContainer" containerID="632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd" Nov 03 22:24:08.671462 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.671443 5272 scope.go:122] "RemoveContainer" containerID="6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7" Nov 03 22:24:08.677553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.677535 5272 scope.go:122] "RemoveContainer" containerID="ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb" Nov 03 22:24:08.677900 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:08.677789 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb\": container with ID starting with ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb not found: ID does not exist" containerID="ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb" Nov 03 22:24:08.677962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.677913 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb"} err="failed to get container status \"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb\": rpc error: code = NotFound desc = could not find container \"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb\": container with ID starting with ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb not found: ID does not exist" Nov 03 22:24:08.678004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.677966 5272 scope.go:122] "RemoveContainer" containerID="f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384" Nov 03 22:24:08.678219 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:08.678203 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384\": container with ID starting with f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384 not found: ID does not exist" containerID="f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384" Nov 03 22:24:08.678261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678226 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384"} err="failed to get container status \"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384\": rpc error: code = NotFound desc = could not find container \"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384\": container with ID starting with f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384 not found: ID does not exist" Nov 03 22:24:08.678261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678241 5272 scope.go:122] "RemoveContainer" containerID="decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec" Nov 03 22:24:08.678465 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:08.678451 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec\": container with ID starting with decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec not found: ID does not exist" containerID="decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec" Nov 03 22:24:08.678513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678468 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec"} err="failed to get container status \"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec\": rpc error: code = NotFound desc = could not find container \"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec\": container with ID starting with decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec not found: ID does not exist" Nov 03 22:24:08.678513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678480 5272 scope.go:122] "RemoveContainer" containerID="f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1" Nov 03 22:24:08.678660 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:08.678645 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1\": container with ID starting with f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1 not found: ID does not exist" containerID="f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1" Nov 03 22:24:08.678695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678665 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1"} err="failed to get container status \"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1\": rpc error: code = NotFound desc = could not find container \"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1\": container with ID starting with f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1 not found: ID does not exist" Nov 03 22:24:08.678695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678679 5272 scope.go:122] "RemoveContainer" containerID="2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2" Nov 03 22:24:08.678897 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:08.678882 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2\": container with ID starting with 2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2 not found: ID does not exist" containerID="2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2" Nov 03 22:24:08.678940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678901 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2"} err="failed to get container status \"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2\": rpc error: code = NotFound desc = could not find container \"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2\": container with ID starting with 2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2 not found: ID does not exist" Nov 03 22:24:08.678940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.678912 5272 scope.go:122] "RemoveContainer" containerID="632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd" Nov 03 22:24:08.679136 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:08.679121 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd\": container with ID starting with 632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd not found: ID does not exist" containerID="632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd" Nov 03 22:24:08.679173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679140 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd"} err="failed to get container status \"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd\": rpc error: code = NotFound desc = could not find container \"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd\": container with ID starting with 632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd not found: ID does not exist" Nov 03 22:24:08.679173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679152 5272 scope.go:122] "RemoveContainer" containerID="6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7" Nov 03 22:24:08.679340 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:08.679326 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7\": container with ID starting with 6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7 not found: ID does not exist" containerID="6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7" Nov 03 22:24:08.679380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679343 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7"} err="failed to get container status \"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7\": rpc error: code = NotFound desc = could not find container \"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7\": container with ID starting with 6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7 not found: ID does not exist" Nov 03 22:24:08.679380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679354 5272 scope.go:122] "RemoveContainer" containerID="ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb" Nov 03 22:24:08.679533 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679518 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb"} err="failed to get container status \"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb\": rpc error: code = NotFound desc = could not find container \"ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb\": container with ID starting with ccded49ed40b43a7f1ba8dd3045dbf04ad2d9c1205e6964df9d9bde518c4e8cb not found: ID does not exist" Nov 03 22:24:08.679573 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679533 5272 scope.go:122] "RemoveContainer" containerID="f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384" Nov 03 22:24:08.679739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679723 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384"} err="failed to get container status \"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384\": rpc error: code = NotFound desc = could not find container \"f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384\": container with ID starting with f74b56d1cbb39bc1b12e50a828d5711664e91f00b2a9593cc860e54e594a8384 not found: ID does not exist" Nov 03 22:24:08.679779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679739 5272 scope.go:122] "RemoveContainer" containerID="decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec" Nov 03 22:24:08.679924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679909 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec"} err="failed to get container status \"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec\": rpc error: code = NotFound desc = could not find container \"decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec\": container with ID starting with decb6c717d09380d24b28ddb1039472a115763d2e60f840f3817b4c36179b2ec not found: ID does not exist" Nov 03 22:24:08.679965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.679924 5272 scope.go:122] "RemoveContainer" containerID="f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1" Nov 03 22:24:08.680106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.680091 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1"} err="failed to get container status \"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1\": rpc error: code = NotFound desc = could not find container \"f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1\": container with ID starting with f00e576ac6e81f380365c49f015c0c12110b1b9ca748fc6e64efc86421b5e0d1 not found: ID does not exist" Nov 03 22:24:08.680145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.680106 5272 scope.go:122] "RemoveContainer" containerID="2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2" Nov 03 22:24:08.680325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.680310 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2"} err="failed to get container status \"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2\": rpc error: code = NotFound desc = could not find container \"2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2\": container with ID starting with 2ff3b0f2dca0a238475061bc3408c37913d11542568ec0f87e6deea54078c3a2 not found: ID does not exist" Nov 03 22:24:08.680325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.680324 5272 scope.go:122] "RemoveContainer" containerID="632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd" Nov 03 22:24:08.680532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.680516 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd"} err="failed to get container status \"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd\": rpc error: code = NotFound desc = could not find container \"632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd\": container with ID starting with 632b53359495ea5ad542ab1e9995c87884b0438747ded249fe00d53d1e6e9acd not found: ID does not exist" Nov 03 22:24:08.680532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.680531 5272 scope.go:122] "RemoveContainer" containerID="6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7" Nov 03 22:24:08.680728 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.680710 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7"} err="failed to get container status \"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7\": rpc error: code = NotFound desc = could not find container \"6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7\": container with ID starting with 6ebada6dab92e1321939276f126e4f9daee760e3865c014e92fb0d7c87bf00a7 not found: ID does not exist" Nov 03 22:24:08.788225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.788201 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:24:08.834561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.834534 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:24:08.834786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.834759 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:24:08.835093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835075 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="init-config-reloader" Nov 03 22:24:08.835142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835094 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="init-config-reloader" Nov 03 22:24:08.835142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835114 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="prom-label-proxy" Nov 03 22:24:08.835142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835120 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="prom-label-proxy" Nov 03 22:24:08.835142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835126 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="alertmanager" Nov 03 22:24:08.835142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835130 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="alertmanager" Nov 03 22:24:08.835142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835137 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-metric" Nov 03 22:24:08.835142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835142 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-metric" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835148 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835153 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835158 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-web" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835163 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-web" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835173 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="config-reloader" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835178 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="config-reloader" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835219 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-web" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835228 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="config-reloader" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835235 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="init-config-reloader" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835241 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="prom-label-proxy" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835246 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="alertmanager" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835251 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy-metric" Nov 03 22:24:08.835339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.835255 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" containerName="kube-rbac-proxy" Nov 03 22:24:08.838969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.838955 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.919697 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.919667 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-web-config\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.919850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.919703 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.919850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.919732 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.919850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.919780 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20049bd4-c939-46bd-9162-5f25c25bc97d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.919924 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20049bd4-c939-46bd-9162-5f25c25bc97d-config-out\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.919965 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20049bd4-c939-46bd-9162-5f25c25bc97d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.920007 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.920025 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.920040 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20049bd4-c939-46bd-9162-5f25c25bc97d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.920055 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prmd\" (UniqueName: \"kubernetes.io/projected/20049bd4-c939-46bd-9162-5f25c25bc97d-kube-api-access-5prmd\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.920120 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-config-volume\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920278 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.920174 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.920278 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.920206 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20049bd4-c939-46bd-9162-5f25c25bc97d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:08.934914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934873 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 22:24:08.934914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934890 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 22:24:08.934914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934900 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 22:24:08.934914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934915 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 22:24:08.935120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934921 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 22:24:08.935120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934902 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mf8dj\"" Nov 03 22:24:08.935120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934880 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 22:24:08.935120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934880 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 22:24:08.935120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.934891 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 22:24:08.935120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.935003 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 22:24:08.947724 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:08.947700 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:24:09.021460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021425 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-web-config\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021465 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021506 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021528 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20049bd4-c939-46bd-9162-5f25c25bc97d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021552 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20049bd4-c939-46bd-9162-5f25c25bc97d-config-out\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021579 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20049bd4-c939-46bd-9162-5f25c25bc97d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021600 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021667 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20049bd4-c939-46bd-9162-5f25c25bc97d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021731 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prmd\" (UniqueName: \"kubernetes.io/projected/20049bd4-c939-46bd-9162-5f25c25bc97d-kube-api-access-5prmd\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.021933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021886 5272 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:24:09.021933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021921 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-config-volume\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.022099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.021977 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.022099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.022023 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20049bd4-c939-46bd-9162-5f25c25bc97d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.022202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.022107 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20049bd4-c939-46bd-9162-5f25c25bc97d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.022673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.022650 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20049bd4-c939-46bd-9162-5f25c25bc97d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.022894 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.022869 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20049bd4-c939-46bd-9162-5f25c25bc97d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.024862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.024821 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-config-volume\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.024862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.024821 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-web-config\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.025239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.025193 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.025239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.025204 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.025397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.025340 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.025457 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.025410 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20049bd4-c939-46bd-9162-5f25c25bc97d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.025599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.025576 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20049bd4-c939-46bd-9162-5f25c25bc97d-config-out\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.025732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.025714 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.026464 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.026444 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20049bd4-c939-46bd-9162-5f25c25bc97d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.079293 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.079267 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prmd\" (UniqueName: \"kubernetes.io/projected/20049bd4-c939-46bd-9162-5f25c25bc97d-kube-api-access-5prmd\") pod \"alertmanager-main-0\" (UID: \"20049bd4-c939-46bd-9162-5f25c25bc97d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.157106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.157047 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 22:24:09.490669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.490596 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a79337-e1e4-40a7-a8cb-5fc3264f9a62" path="/var/lib/kubelet/pods/c0a79337-e1e4-40a7-a8cb-5fc3264f9a62/volumes" Nov 03 22:24:09.584652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.584624 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 22:24:09.592399 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:24:09.592372 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20049bd4_c939_46bd_9162_5f25c25bc97d.slice/crio-338fb5210595d2ee79ca018e984f26e1a5fca5a983a87740a994ff2e6b7fcb9b WatchSource:0}: Error finding container 338fb5210595d2ee79ca018e984f26e1a5fca5a983a87740a994ff2e6b7fcb9b: Status 404 returned error can't find the container with id 338fb5210595d2ee79ca018e984f26e1a5fca5a983a87740a994ff2e6b7fcb9b Nov 03 22:24:09.635025 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:09.634996 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerStarted","Data":"338fb5210595d2ee79ca018e984f26e1a5fca5a983a87740a994ff2e6b7fcb9b"} Nov 03 22:24:10.640750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:10.640710 5272 generic.go:356] "Generic (PLEG): container finished" podID="20049bd4-c939-46bd-9162-5f25c25bc97d" containerID="bae68ec7fc2c04392d3c583434ad93f4fb571692b52b229365f25615c8e9a477" exitCode=0 Nov 03 22:24:10.641219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:10.640782 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerDied","Data":"bae68ec7fc2c04392d3c583434ad93f4fb571692b52b229365f25615c8e9a477"} Nov 03 22:24:11.648017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:11.647980 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerStarted","Data":"21fff2a67179d8659cca1d32b8406053668076cbf4a11c90dfd79be080a6642b"} Nov 03 22:24:11.648557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:11.648043 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerStarted","Data":"acbf0d8c9c1672ad635ceb26f18d9bd9a280598c0bbccf79a2dcf77d44322748"} Nov 03 22:24:11.648557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:11.648068 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerStarted","Data":"5215c1fda6e6b319d85eb1ceb6bf765eee8c42f7a168607d220d91422ceef9ff"} Nov 03 22:24:11.648557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:11.648093 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerStarted","Data":"c717eeffc0a14a3a93ee5962e34cbf11776d3972f7fb56b495344a314ccf5eb8"} Nov 03 22:24:11.648557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:11.648117 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerStarted","Data":"aacbbfc01c94798e62e63f7850035abc1ddc2ce5a274afb9bb2394020bb2b508"} Nov 03 22:24:11.648557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:11.648139 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049bd4-c939-46bd-9162-5f25c25bc97d","Type":"ContainerStarted","Data":"2fbf7036c55553f5fbde6c81f422a60f8af427a71f4fca84c2e907a0d858a79d"} Nov 03 22:24:11.771233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:11.771181 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.760253496 podStartE2EDuration="3.771168522s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.915026ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:11.770526477 +0000 UTC m=+32.392834786" watchObservedRunningTime="2026-11-03 22:24:11.771168522 +0000 UTC m=+32.393476827" Nov 03 22:24:16.892473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:24:16.892438 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hdq68" Nov 03 22:24:24.498394 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:24.498357 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:24:54.498483 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:24:54.498446 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:25:24.498588 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:25:24.498506 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:25:49.458887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.458852 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx"] Nov 03 22:25:49.461317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.461300 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.490856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.490826 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.490856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.490856 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.491040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.490953 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6kwg\" (UniqueName: \"kubernetes.io/projected/0af9f103-db32-468e-b623-8f9af0e1481c-kube-api-access-n6kwg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.556340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.556309 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 22:25:49.556340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.556330 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 22:25:49.556541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.556421 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vpwbg\"" Nov 03 22:25:49.556904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.556886 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx"] Nov 03 22:25:49.558980 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.558964 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw"] Nov 03 22:25:49.561373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.561359 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.591418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591394 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6kwg\" (UniqueName: \"kubernetes.io/projected/0af9f103-db32-468e-b623-8f9af0e1481c-kube-api-access-n6kwg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.591565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591433 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.591565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591453 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.591565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591540 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.591689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591591 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v8lx\" (UniqueName: \"kubernetes.io/projected/fb489906-b653-4685-bc24-8422395f987e-kube-api-access-8v8lx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.591736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591675 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.591989 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591970 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.591989 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.591981 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.612642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.612608 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-h5xqf\"" Nov 03 22:25:49.622337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.622311 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw"] Nov 03 22:25:49.648724 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.648688 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6kwg\" (UniqueName: \"kubernetes.io/projected/0af9f103-db32-468e-b623-8f9af0e1481c-kube-api-access-n6kwg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.692919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.692891 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.693061 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.692930 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v8lx\" (UniqueName: \"kubernetes.io/projected/fb489906-b653-4685-bc24-8422395f987e-kube-api-access-8v8lx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.693061 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.692957 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.693294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.693275 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.693367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.693318 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.750644 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.750580 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v8lx\" (UniqueName: \"kubernetes.io/projected/fb489906-b653-4685-bc24-8422395f987e-kube-api-access-8v8lx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:49.774897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.774869 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:25:49.874897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:49.874864 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:25:50.143483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.143454 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx"] Nov 03 22:25:50.146205 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.146183 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j"] Nov 03 22:25:50.149904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.149882 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.152461 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:25:50.152440 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af9f103_db32_468e_b623_8f9af0e1481c.slice/crio-3fc0b881184313fd5acf2a1ee844d02a148d55fa7f7201f1f3b18b2b4b14cebb WatchSource:0}: Error finding container 3fc0b881184313fd5acf2a1ee844d02a148d55fa7f7201f1f3b18b2b4b14cebb: Status 404 returned error can't find the container with id 3fc0b881184313fd5acf2a1ee844d02a148d55fa7f7201f1f3b18b2b4b14cebb Nov 03 22:25:50.154480 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.154466 5272 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:25:50.196156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.196123 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.196317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.196168 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f3c6da-bee7-4227-8140-4a5700282853-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.196317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.196192 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq26v\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-kube-api-access-tq26v\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.196317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.196217 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.196317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.196297 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.196520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.196402 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.196520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.196452 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.202281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.202257 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:25:50.202389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.202287 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:25:50.202389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.202319 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:25:50.202597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.202584 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:25:50.202634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.202595 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8qdxr\"" Nov 03 22:25:50.202634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.202602 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:25:50.209917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.209892 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j"] Nov 03 22:25:50.260546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.260515 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw"] Nov 03 22:25:50.268871 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:25:50.268817 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb489906_b653_4685_bc24_8422395f987e.slice/crio-399d8e093c5819aebbcfffcc595dbc0733aa3e628e1f89033431f6da93655f1b WatchSource:0}: Error finding container 399d8e093c5819aebbcfffcc595dbc0733aa3e628e1f89033431f6da93655f1b: Status 404 returned error can't find the container with id 399d8e093c5819aebbcfffcc595dbc0733aa3e628e1f89033431f6da93655f1b Nov 03 22:25:50.297198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297176 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq26v\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-kube-api-access-tq26v\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.297307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297211 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.297307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297231 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.297307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297259 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.297307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297293 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.297482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297332 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.297482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297374 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f3c6da-bee7-4227-8140-4a5700282853-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.298006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.297978 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.299615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.299595 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.299708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.299621 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.299876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.299860 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f3c6da-bee7-4227-8140-4a5700282853-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.299927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.299912 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.302851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.302835 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:25:50.354728 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.354697 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.354890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.354858 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq26v\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-kube-api-access-tq26v\") pod \"istiod-openshift-gateway-7876bfcb74-tbw6j\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.462193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.462116 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:50.880666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.880631 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j"] Nov 03 22:25:50.888395 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:25:50.888367 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f3c6da_bee7_4227_8140_4a5700282853.slice/crio-ec38d864f9e98f9d3cd9fc00ce1f44a9c024c3efbcd4144b6027146d57076708 WatchSource:0}: Error finding container ec38d864f9e98f9d3cd9fc00ce1f44a9c024c3efbcd4144b6027146d57076708: Status 404 returned error can't find the container with id ec38d864f9e98f9d3cd9fc00ce1f44a9c024c3efbcd4144b6027146d57076708 Nov 03 22:25:50.941237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.941200 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" event={"ID":"75f3c6da-bee7-4227-8140-4a5700282853","Type":"ContainerStarted","Data":"ec38d864f9e98f9d3cd9fc00ce1f44a9c024c3efbcd4144b6027146d57076708"} Nov 03 22:25:50.942327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.942298 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" event={"ID":"fb489906-b653-4685-bc24-8422395f987e","Type":"ContainerStarted","Data":"399d8e093c5819aebbcfffcc595dbc0733aa3e628e1f89033431f6da93655f1b"} Nov 03 22:25:50.943341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:50.943320 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" event={"ID":"0af9f103-db32-468e-b623-8f9af0e1481c","Type":"ContainerStarted","Data":"3fc0b881184313fd5acf2a1ee844d02a148d55fa7f7201f1f3b18b2b4b14cebb"} Nov 03 22:25:54.514786 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:25:54.514743 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:25:55.705175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:55.705139 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:25:55.705560 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:55.705218 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:25:55.966033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:55.965948 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" event={"ID":"75f3c6da-bee7-4227-8140-4a5700282853","Type":"ContainerStarted","Data":"1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145"} Nov 03 22:25:55.966193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:55.966090 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:25:56.172973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:56.172583 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" podStartSLOduration=1.358750506 podStartE2EDuration="6.172565124s" totalImagesPullingTime="4.813814618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:56.125831718 +0000 UTC m=+136.748140027" watchObservedRunningTime="2026-11-03 22:25:56.172565124 +0000 UTC m=+136.794873433" Nov 03 22:25:56.973534 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:56.973499 5272 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-tbw6j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:25:56.973959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:56.973561 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" podUID="75f3c6da-bee7-4227-8140-4a5700282853" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:25:57.977103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:57.977018 5272 generic.go:356] "Generic (PLEG): container finished" podID="fb489906-b653-4685-bc24-8422395f987e" containerID="f2d242b0d6836b9d906b6d528c9cf0f4af16fc7d538c2ae98ea6473cf6662b20" exitCode=0 Nov 03 22:25:57.977558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:57.977104 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" event={"ID":"fb489906-b653-4685-bc24-8422395f987e","Type":"ContainerDied","Data":"f2d242b0d6836b9d906b6d528c9cf0f4af16fc7d538c2ae98ea6473cf6662b20"} Nov 03 22:25:57.978583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:57.978557 5272 generic.go:356] "Generic (PLEG): container finished" podID="0af9f103-db32-468e-b623-8f9af0e1481c" containerID="3cbc2db64aa2a4468dc0ab5c2cb9385f7057ded979fbf79d4b45c28850502624" exitCode=0 Nov 03 22:25:57.978691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:57.978618 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" event={"ID":"0af9f103-db32-468e-b623-8f9af0e1481c","Type":"ContainerDied","Data":"3cbc2db64aa2a4468dc0ab5c2cb9385f7057ded979fbf79d4b45c28850502624"} Nov 03 22:25:59.972875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:25:59.972843 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:26:02.996552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:02.996520 5272 generic.go:356] "Generic (PLEG): container finished" podID="fb489906-b653-4685-bc24-8422395f987e" containerID="c8785935ad4cd87978d3800dc695b2f918236af0272d6550b1157ec675c40679" exitCode=0 Nov 03 22:26:02.996952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:02.996586 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" event={"ID":"fb489906-b653-4685-bc24-8422395f987e","Type":"ContainerDied","Data":"c8785935ad4cd87978d3800dc695b2f918236af0272d6550b1157ec675c40679"} Nov 03 22:26:10.024888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:10.024851 5272 generic.go:356] "Generic (PLEG): container finished" podID="0af9f103-db32-468e-b623-8f9af0e1481c" containerID="9e9faf0fbdb575589a212de99311834ce05fa11d695f3aa4c135075f2d05604b" exitCode=0 Nov 03 22:26:10.025333 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:10.024930 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" event={"ID":"0af9f103-db32-468e-b623-8f9af0e1481c","Type":"ContainerDied","Data":"9e9faf0fbdb575589a212de99311834ce05fa11d695f3aa4c135075f2d05604b"} Nov 03 22:26:11.030857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:11.030821 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" event={"ID":"fb489906-b653-4685-bc24-8422395f987e","Type":"ContainerStarted","Data":"653e07b95f243b633ce25046cbd5e99fce41be62d8b6771a43b33305541f82d9"} Nov 03 22:26:11.142245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:11.142198 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" podStartSLOduration=3.30746333 podStartE2EDuration="22.142181603s" totalImagesPullingTime="18.834718273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:11.141613876 +0000 UTC m=+151.763922184" watchObservedRunningTime="2026-11-03 22:26:11.142181603 +0000 UTC m=+151.764489911" Nov 03 22:26:12.036269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:12.036236 5272 generic.go:356] "Generic (PLEG): container finished" podID="fb489906-b653-4685-bc24-8422395f987e" containerID="653e07b95f243b633ce25046cbd5e99fce41be62d8b6771a43b33305541f82d9" exitCode=0 Nov 03 22:26:12.036700 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:12.036313 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" event={"ID":"fb489906-b653-4685-bc24-8422395f987e","Type":"ContainerDied","Data":"653e07b95f243b633ce25046cbd5e99fce41be62d8b6771a43b33305541f82d9"} Nov 03 22:26:12.038250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:12.038229 5272 generic.go:356] "Generic (PLEG): container finished" podID="0af9f103-db32-468e-b623-8f9af0e1481c" containerID="3e0958838102f30d0fd923089e846eadc72237dda2ee93794487d50b50a4c8db" exitCode=0 Nov 03 22:26:12.038250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:12.038256 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" event={"ID":"0af9f103-db32-468e-b623-8f9af0e1481c","Type":"ContainerDied","Data":"3e0958838102f30d0fd923089e846eadc72237dda2ee93794487d50b50a4c8db"} Nov 03 22:26:13.179858 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.179830 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:26:13.203983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.203956 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:26:13.286755 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.286724 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-bundle\") pod \"fb489906-b653-4685-bc24-8422395f987e\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " Nov 03 22:26:13.286923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.286766 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-bundle\") pod \"0af9f103-db32-468e-b623-8f9af0e1481c\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " Nov 03 22:26:13.286923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.286845 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af9f103-db32-468e-b623-8f9af0e1481c-kube-api-access-n6kwg\" (UniqueName: \"kubernetes.io/projected/0af9f103-db32-468e-b623-8f9af0e1481c-kube-api-access-n6kwg\") pod \"0af9f103-db32-468e-b623-8f9af0e1481c\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " Nov 03 22:26:13.286923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.286881 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-util\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-util\") pod \"fb489906-b653-4685-bc24-8422395f987e\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " Nov 03 22:26:13.286923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.286911 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb489906-b653-4685-bc24-8422395f987e-kube-api-access-8v8lx\" (UniqueName: \"kubernetes.io/projected/fb489906-b653-4685-bc24-8422395f987e-kube-api-access-8v8lx\") pod \"fb489906-b653-4685-bc24-8422395f987e\" (UID: \"fb489906-b653-4685-bc24-8422395f987e\") " Nov 03 22:26:13.287110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.287037 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-util\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-util\") pod \"0af9f103-db32-468e-b623-8f9af0e1481c\" (UID: \"0af9f103-db32-468e-b623-8f9af0e1481c\") " Nov 03 22:26:13.287327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.287279 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-bundle" pod "0af9f103-db32-468e-b623-8f9af0e1481c" (UID: "0af9f103-db32-468e-b623-8f9af0e1481c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:26:13.287574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.287558 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-bundle" pod "fb489906-b653-4685-bc24-8422395f987e" (UID: "fb489906-b653-4685-bc24-8422395f987e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:26:13.289184 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.289149 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af9f103-db32-468e-b623-8f9af0e1481c-kube-api-access-n6kwg" pod "0af9f103-db32-468e-b623-8f9af0e1481c" (UID: "0af9f103-db32-468e-b623-8f9af0e1481c"). InnerVolumeSpecName "kube-api-access-n6kwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:26:13.289290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.289265 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb489906-b653-4685-bc24-8422395f987e-kube-api-access-8v8lx" pod "fb489906-b653-4685-bc24-8422395f987e" (UID: "fb489906-b653-4685-bc24-8422395f987e"). InnerVolumeSpecName "kube-api-access-8v8lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:26:13.292097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.292049 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-util" pod "fb489906-b653-4685-bc24-8422395f987e" (UID: "fb489906-b653-4685-bc24-8422395f987e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:26:13.292525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.292359 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-util" pod "0af9f103-db32-468e-b623-8f9af0e1481c" (UID: "0af9f103-db32-468e-b623-8f9af0e1481c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:26:13.387883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.387842 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6kwg\" (UniqueName: \"kubernetes.io/projected/0af9f103-db32-468e-b623-8f9af0e1481c-kube-api-access-n6kwg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:26:13.387883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.387877 5272 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-util\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:26:13.387883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.387890 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v8lx\" (UniqueName: \"kubernetes.io/projected/fb489906-b653-4685-bc24-8422395f987e-kube-api-access-8v8lx\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:26:13.388124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.387902 5272 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-util\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:26:13.388124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.387915 5272 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb489906-b653-4685-bc24-8422395f987e-bundle\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:26:13.388124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:13.387927 5272 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0af9f103-db32-468e-b623-8f9af0e1481c-bundle\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:26:14.046959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:14.046928 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" Nov 03 22:26:14.046959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:14.046951 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s9jfx" event={"ID":"0af9f103-db32-468e-b623-8f9af0e1481c","Type":"ContainerDied","Data":"3fc0b881184313fd5acf2a1ee844d02a148d55fa7f7201f1f3b18b2b4b14cebb"} Nov 03 22:26:14.047211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:14.047005 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc0b881184313fd5acf2a1ee844d02a148d55fa7f7201f1f3b18b2b4b14cebb" Nov 03 22:26:14.049008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:14.048964 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" event={"ID":"fb489906-b653-4685-bc24-8422395f987e","Type":"ContainerDied","Data":"399d8e093c5819aebbcfffcc595dbc0733aa3e628e1f89033431f6da93655f1b"} Nov 03 22:26:14.049008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:14.049004 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="399d8e093c5819aebbcfffcc595dbc0733aa3e628e1f89033431f6da93655f1b" Nov 03 22:26:14.049164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:14.049023 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4twpw" Nov 03 22:26:24.499133 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:26:24.499095 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:26:31.203927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.203885 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9"] Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204243 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fb489906-b653-4685-bc24-8422395f987e" containerName="util" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204259 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb489906-b653-4685-bc24-8422395f987e" containerName="util" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204270 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fb489906-b653-4685-bc24-8422395f987e" containerName="pull" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204278 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb489906-b653-4685-bc24-8422395f987e" containerName="pull" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204293 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0af9f103-db32-468e-b623-8f9af0e1481c" containerName="extract" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204300 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af9f103-db32-468e-b623-8f9af0e1481c" containerName="extract" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204312 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fb489906-b653-4685-bc24-8422395f987e" containerName="extract" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204318 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb489906-b653-4685-bc24-8422395f987e" containerName="extract" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204332 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0af9f103-db32-468e-b623-8f9af0e1481c" containerName="pull" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204339 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af9f103-db32-468e-b623-8f9af0e1481c" containerName="pull" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204366 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0af9f103-db32-468e-b623-8f9af0e1481c" containerName="util" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204372 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af9f103-db32-468e-b623-8f9af0e1481c" containerName="util" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204432 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af9f103-db32-468e-b623-8f9af0e1481c" containerName="extract" Nov 03 22:26:31.204435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.204443 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb489906-b653-4685-bc24-8422395f987e" containerName="extract" Nov 03 22:26:31.206893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.206871 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.303094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.303058 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:26:31.303271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.303112 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6xv29\"" Nov 03 22:26:31.303271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.303111 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:26:31.303271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.303132 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:26:31.303271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.303139 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:26:31.303271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.303187 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:26:31.303944 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.303922 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9"] Nov 03 22:26:31.310540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.310517 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kkvv\" (UniqueName: \"kubernetes.io/projected/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-kube-api-access-6kkvv\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.310649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.310569 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.310649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.310595 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-tmp\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.310737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.310671 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.310852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.310836 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-metrics-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.310917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.310870 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-manifests\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.411457 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.411429 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.411642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.411487 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-metrics-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.411642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.411517 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-manifests\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.411642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.411558 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kkvv\" (UniqueName: \"kubernetes.io/projected/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-kube-api-access-6kkvv\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.411642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.411602 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.411642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.411622 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-tmp\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.412184 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.412089 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-manifests\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.413910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.413890 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-tmp\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.414216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.414194 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-metrics-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.414503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.414483 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.414539 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.414483 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.470577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.470520 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kkvv\" (UniqueName: \"kubernetes.io/projected/7b42ccd9-6f7c-420b-9721-5fd16cfaded6-kube-api-access-6kkvv\") pod \"opendatahub-operator-controller-manager-8bc866f6-k2pd9\" (UID: \"7b42ccd9-6f7c-420b-9721-5fd16cfaded6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:31.524116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:31.524090 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:32.022186 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:26:32.022161 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b42ccd9_6f7c_420b_9721_5fd16cfaded6.slice/crio-bf11189add23f9f38dc0ef78f7e87eb8512d519a674ae79ab5d21fd08a2b4b90 WatchSource:0}: Error finding container bf11189add23f9f38dc0ef78f7e87eb8512d519a674ae79ab5d21fd08a2b4b90: Status 404 returned error can't find the container with id bf11189add23f9f38dc0ef78f7e87eb8512d519a674ae79ab5d21fd08a2b4b90 Nov 03 22:26:32.060742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:32.060710 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9"] Nov 03 22:26:32.113508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:32.113470 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" event={"ID":"7b42ccd9-6f7c-420b-9721-5fd16cfaded6","Type":"ContainerStarted","Data":"bf11189add23f9f38dc0ef78f7e87eb8512d519a674ae79ab5d21fd08a2b4b90"} Nov 03 22:26:36.129059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:36.129026 5272 generic.go:356] "Generic (PLEG): container finished" podID="7b42ccd9-6f7c-420b-9721-5fd16cfaded6" containerID="7e65ea5b9740f52ab2ce6b7b58b132e30c521922966d0c793da22fa5a2333730" exitCode=0 Nov 03 22:26:36.129421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:36.129064 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" event={"ID":"7b42ccd9-6f7c-420b-9721-5fd16cfaded6","Type":"ContainerDied","Data":"7e65ea5b9740f52ab2ce6b7b58b132e30c521922966d0c793da22fa5a2333730"} Nov 03 22:26:37.135065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:37.135024 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" event={"ID":"7b42ccd9-6f7c-420b-9721-5fd16cfaded6","Type":"ContainerStarted","Data":"ffb9858ca7cf4432fee8702dca59284e494d9b15246ffa7aed7f6ee36f5caa37"} Nov 03 22:26:37.135478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:37.135160 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:37.361823 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:37.361700 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" podStartSLOduration=1.48931506 podStartE2EDuration="6.361684114s" totalImagesPullingTime="4.4571207s" totalInitContainerRuntime="415.248354ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:37.315326209 +0000 UTC m=+177.937634518" watchObservedRunningTime="2026-11-03 22:26:37.361684114 +0000 UTC m=+177.983992422" Nov 03 22:26:48.140572 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:26:48.140543 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8bc866f6-k2pd9" Nov 03 22:26:54.499751 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:26:54.499675 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:27:03.288287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.288255 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84cff947f5-6bglq"] Nov 03 22:27:03.290908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.290882 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.370192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.370152 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/204b7f88-d17a-4326-a5cf-9ef201d58871-tmp\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.370192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.370193 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgkg\" (UniqueName: \"kubernetes.io/projected/204b7f88-d17a-4326-a5cf-9ef201d58871-kube-api-access-cwgkg\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.370451 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.370227 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204b7f88-d17a-4326-a5cf-9ef201d58871-tls-certs\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.386377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.386349 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 22:27:03.386377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.386360 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j4bnj\"" Nov 03 22:27:03.386377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.386362 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 22:27:03.387764 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.387747 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84cff947f5-6bglq"] Nov 03 22:27:03.471166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.471137 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/204b7f88-d17a-4326-a5cf-9ef201d58871-tmp\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.471327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.471174 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgkg\" (UniqueName: \"kubernetes.io/projected/204b7f88-d17a-4326-a5cf-9ef201d58871-kube-api-access-cwgkg\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.471327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.471204 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204b7f88-d17a-4326-a5cf-9ef201d58871-tls-certs\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.473408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.473380 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/204b7f88-d17a-4326-a5cf-9ef201d58871-tmp\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.473637 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.473618 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204b7f88-d17a-4326-a5cf-9ef201d58871-tls-certs\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.531066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.531028 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgkg\" (UniqueName: \"kubernetes.io/projected/204b7f88-d17a-4326-a5cf-9ef201d58871-kube-api-access-cwgkg\") pod \"kube-auth-proxy-84cff947f5-6bglq\" (UID: \"204b7f88-d17a-4326-a5cf-9ef201d58871\") " pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.604006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.603931 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" Nov 03 22:27:03.978347 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:03.978251 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84cff947f5-6bglq"] Nov 03 22:27:03.984014 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:27:03.983979 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204b7f88_d17a_4326_a5cf_9ef201d58871.slice/crio-5f4f10e6c51d601cbfffba3e95444bdf551740b7e38fe171baac5a942577658e WatchSource:0}: Error finding container 5f4f10e6c51d601cbfffba3e95444bdf551740b7e38fe171baac5a942577658e: Status 404 returned error can't find the container with id 5f4f10e6c51d601cbfffba3e95444bdf551740b7e38fe171baac5a942577658e Nov 03 22:27:04.234173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:04.234096 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" event={"ID":"204b7f88-d17a-4326-a5cf-9ef201d58871","Type":"ContainerStarted","Data":"5f4f10e6c51d601cbfffba3e95444bdf551740b7e38fe171baac5a942577658e"} Nov 03 22:27:08.233395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.233365 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-pl5tf"] Nov 03 22:27:08.235556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.235539 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.249924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.249898 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" event={"ID":"204b7f88-d17a-4326-a5cf-9ef201d58871","Type":"ContainerStarted","Data":"4ef54daaed0687b5c2eb59e6a4ba31d35cd3a6c46459c2048751247dc06705ac"} Nov 03 22:27:08.310930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.310898 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wfn\" (UniqueName: \"kubernetes.io/projected/a375e442-512d-4a83-922c-e1096a1c08ff-kube-api-access-p6wfn\") pod \"cert-manager-597b95cb84-pl5tf\" (UID: \"a375e442-512d-4a83-922c-e1096a1c08ff\") " pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.311090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.311029 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a375e442-512d-4a83-922c-e1096a1c08ff-bound-sa-token\") pod \"cert-manager-597b95cb84-pl5tf\" (UID: \"a375e442-512d-4a83-922c-e1096a1c08ff\") " pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.330598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.330567 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-frl68\"" Nov 03 22:27:08.330743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.330634 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 22:27:08.330743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.330633 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 22:27:08.331424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.331407 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-pl5tf"] Nov 03 22:27:08.415690 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.412103 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a375e442-512d-4a83-922c-e1096a1c08ff-bound-sa-token\") pod \"cert-manager-597b95cb84-pl5tf\" (UID: \"a375e442-512d-4a83-922c-e1096a1c08ff\") " pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.415690 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.412197 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wfn\" (UniqueName: \"kubernetes.io/projected/a375e442-512d-4a83-922c-e1096a1c08ff-kube-api-access-p6wfn\") pod \"cert-manager-597b95cb84-pl5tf\" (UID: \"a375e442-512d-4a83-922c-e1096a1c08ff\") " pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.451894 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.451843 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84cff947f5-6bglq" podStartSLOduration=2.004262291 podStartE2EDuration="5.451829809s" totalImagesPullingTime="3.447567518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:08.45020653 +0000 UTC m=+209.072514838" watchObservedRunningTime="2026-11-03 22:27:08.451829809 +0000 UTC m=+209.074138107" Nov 03 22:27:08.470789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.470755 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a375e442-512d-4a83-922c-e1096a1c08ff-bound-sa-token\") pod \"cert-manager-597b95cb84-pl5tf\" (UID: \"a375e442-512d-4a83-922c-e1096a1c08ff\") " pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.470975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.470909 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wfn\" (UniqueName: \"kubernetes.io/projected/a375e442-512d-4a83-922c-e1096a1c08ff-kube-api-access-p6wfn\") pod \"cert-manager-597b95cb84-pl5tf\" (UID: \"a375e442-512d-4a83-922c-e1096a1c08ff\") " pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.557102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.557072 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-pl5tf" Nov 03 22:27:08.927177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:08.927102 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-pl5tf"] Nov 03 22:27:08.934172 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:27:08.934145 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda375e442_512d_4a83_922c_e1096a1c08ff.slice/crio-aa4a36374f17c658c59163da040848c412098afe49920075f3c8caf0452fd7ef WatchSource:0}: Error finding container aa4a36374f17c658c59163da040848c412098afe49920075f3c8caf0452fd7ef: Status 404 returned error can't find the container with id aa4a36374f17c658c59163da040848c412098afe49920075f3c8caf0452fd7ef Nov 03 22:27:09.254464 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:09.254377 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-pl5tf" event={"ID":"a375e442-512d-4a83-922c-e1096a1c08ff","Type":"ContainerStarted","Data":"aa4a36374f17c658c59163da040848c412098afe49920075f3c8caf0452fd7ef"} Nov 03 22:27:13.273575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:13.273531 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-pl5tf" event={"ID":"a375e442-512d-4a83-922c-e1096a1c08ff","Type":"ContainerStarted","Data":"28c949b10c3d0c6489f1e3ecc45dee0e2105371b276591130882a9cd42aff0f8"} Nov 03 22:27:13.387744 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:13.387680 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-pl5tf" podStartSLOduration=1.769532682 podStartE2EDuration="5.387661503s" totalImagesPullingTime="3.618128821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:13.386460775 +0000 UTC m=+214.008769082" watchObservedRunningTime="2026-11-03 22:27:13.387661503 +0000 UTC m=+214.009969810" Nov 03 22:27:21.385209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.385169 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m45c8"] Nov 03 22:27:21.387651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.387626 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m45c8" Nov 03 22:27:21.436865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.436827 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:27:21.482733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.482704 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zkzg4\"" Nov 03 22:27:21.482733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.482725 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:27:21.483651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.483627 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m45c8"] Nov 03 22:27:21.514646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.514612 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qmn\" (UniqueName: \"kubernetes.io/projected/2c63a1e0-f9c6-4010-bdf9-45e1e567492e-kube-api-access-z7qmn\") pod \"authorino-57f7f8659f-m45c8\" (UID: \"2c63a1e0-f9c6-4010-bdf9-45e1e567492e\") " pod="kuadrant-system/authorino-57f7f8659f-m45c8" Nov 03 22:27:21.615471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.615439 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qmn\" (UniqueName: \"kubernetes.io/projected/2c63a1e0-f9c6-4010-bdf9-45e1e567492e-kube-api-access-z7qmn\") pod \"authorino-57f7f8659f-m45c8\" (UID: \"2c63a1e0-f9c6-4010-bdf9-45e1e567492e\") " pod="kuadrant-system/authorino-57f7f8659f-m45c8" Nov 03 22:27:21.672918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.672843 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qmn\" (UniqueName: \"kubernetes.io/projected/2c63a1e0-f9c6-4010-bdf9-45e1e567492e-kube-api-access-z7qmn\") pod \"authorino-57f7f8659f-m45c8\" (UID: \"2c63a1e0-f9c6-4010-bdf9-45e1e567492e\") " pod="kuadrant-system/authorino-57f7f8659f-m45c8" Nov 03 22:27:21.698615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:21.698581 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m45c8" Nov 03 22:27:22.062917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:22.062882 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m45c8"] Nov 03 22:27:22.070771 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:27:22.070745 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c63a1e0_f9c6_4010_bdf9_45e1e567492e.slice/crio-6a1baeca426cc1c9195dfcdb851f7c8168d48f9e9bfeefb263256d74ccac5b17 WatchSource:0}: Error finding container 6a1baeca426cc1c9195dfcdb851f7c8168d48f9e9bfeefb263256d74ccac5b17: Status 404 returned error can't find the container with id 6a1baeca426cc1c9195dfcdb851f7c8168d48f9e9bfeefb263256d74ccac5b17 Nov 03 22:27:22.306972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:22.306936 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m45c8" event={"ID":"2c63a1e0-f9c6-4010-bdf9-45e1e567492e","Type":"ContainerStarted","Data":"6a1baeca426cc1c9195dfcdb851f7c8168d48f9e9bfeefb263256d74ccac5b17"} Nov 03 22:27:24.505877 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:27:24.505837 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:27:26.324074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:26.324042 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m45c8" event={"ID":"2c63a1e0-f9c6-4010-bdf9-45e1e567492e","Type":"ContainerStarted","Data":"1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885"} Nov 03 22:27:26.436821 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:26.434346 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-m45c8" podStartSLOduration=1.330425644 podStartE2EDuration="5.434328047s" totalImagesPullingTime="4.103902403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:26.432102893 +0000 UTC m=+227.054411199" watchObservedRunningTime="2026-11-03 22:27:26.434328047 +0000 UTC m=+227.056636355" Nov 03 22:27:45.672820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.672771 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ql7vl"] Nov 03 22:27:45.676780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.676765 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" Nov 03 22:27:45.802381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.802350 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ql7vl"] Nov 03 22:27:45.822413 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.822391 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxz7\" (UniqueName: \"kubernetes.io/projected/1d7be776-2c84-4863-872f-c511231562e3-kube-api-access-bbxz7\") pod \"authorino-56c6c9ddd5-ql7vl\" (UID: \"1d7be776-2c84-4863-872f-c511231562e3\") " pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" Nov 03 22:27:45.861069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.861047 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b"] Nov 03 22:27:45.865647 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.865620 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:45.917591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.917568 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 22:27:45.917886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.917859 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5plgf\"" Nov 03 22:27:45.918018 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.918003 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 22:27:45.918907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.918879 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b"] Nov 03 22:27:45.923586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.923528 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxz7\" (UniqueName: \"kubernetes.io/projected/1d7be776-2c84-4863-872f-c511231562e3-kube-api-access-bbxz7\") pod \"authorino-56c6c9ddd5-ql7vl\" (UID: \"1d7be776-2c84-4863-872f-c511231562e3\") " pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" Nov 03 22:27:45.983213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.983187 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxz7\" (UniqueName: \"kubernetes.io/projected/1d7be776-2c84-4863-872f-c511231562e3-kube-api-access-bbxz7\") pod \"authorino-56c6c9ddd5-ql7vl\" (UID: \"1d7be776-2c84-4863-872f-c511231562e3\") " pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" Nov 03 22:27:45.986882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:45.986865 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" Nov 03 22:27:46.024433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.024405 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c84240c-84ab-43b3-ba71-1235367c0987-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.024548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.024437 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcdt\" (UniqueName: \"kubernetes.io/projected/9c84240c-84ab-43b3-ba71-1235367c0987-kube-api-access-cbcdt\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.024548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.024462 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c84240c-84ab-43b3-ba71-1235367c0987-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.125708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.125678 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c84240c-84ab-43b3-ba71-1235367c0987-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.125869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.125751 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c84240c-84ab-43b3-ba71-1235367c0987-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.125869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.125772 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcdt\" (UniqueName: \"kubernetes.io/projected/9c84240c-84ab-43b3-ba71-1235367c0987-kube-api-access-cbcdt\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.126455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.126429 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c84240c-84ab-43b3-ba71-1235367c0987-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.128050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.128029 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c84240c-84ab-43b3-ba71-1235367c0987-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.184125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.184067 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcdt\" (UniqueName: \"kubernetes.io/projected/9c84240c-84ab-43b3-ba71-1235367c0987-kube-api-access-cbcdt\") pod \"kuadrant-console-plugin-66c4fb575d-8p92b\" (UID: \"9c84240c-84ab-43b3-ba71-1235367c0987\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.361184 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.361147 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ql7vl"] Nov 03 22:27:46.369069 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:27:46.369034 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d7be776_2c84_4863_872f_c511231562e3.slice/crio-def5b16b5962678ad949933ed2dccf7ee307603f17392f36cd61a2fd0535f1ed WatchSource:0}: Error finding container def5b16b5962678ad949933ed2dccf7ee307603f17392f36cd61a2fd0535f1ed: Status 404 returned error can't find the container with id def5b16b5962678ad949933ed2dccf7ee307603f17392f36cd61a2fd0535f1ed Nov 03 22:27:46.397201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.397167 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" event={"ID":"1d7be776-2c84-4863-872f-c511231562e3","Type":"ContainerStarted","Data":"def5b16b5962678ad949933ed2dccf7ee307603f17392f36cd61a2fd0535f1ed"} Nov 03 22:27:46.477064 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.477010 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" Nov 03 22:27:46.843055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:46.843026 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b"] Nov 03 22:27:46.850614 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:27:46.850587 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c84240c_84ab_43b3_ba71_1235367c0987.slice/crio-2df6f14d7827c0a7142bde582444095e5806ad3fabc9f9cc5f4b39ec067ee883 WatchSource:0}: Error finding container 2df6f14d7827c0a7142bde582444095e5806ad3fabc9f9cc5f4b39ec067ee883: Status 404 returned error can't find the container with id 2df6f14d7827c0a7142bde582444095e5806ad3fabc9f9cc5f4b39ec067ee883 Nov 03 22:27:47.401818 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:47.401766 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" event={"ID":"9c84240c-84ab-43b3-ba71-1235367c0987","Type":"ContainerStarted","Data":"2df6f14d7827c0a7142bde582444095e5806ad3fabc9f9cc5f4b39ec067ee883"} Nov 03 22:27:48.406996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:48.406959 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" event={"ID":"1d7be776-2c84-4863-872f-c511231562e3","Type":"ContainerStarted","Data":"3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd"} Nov 03 22:27:48.519695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:48.519649 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" podStartSLOduration=2.053979443 podStartE2EDuration="3.519634254s" totalImagesPullingTime="1.465654811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:48.518971716 +0000 UTC m=+249.141280023" watchObservedRunningTime="2026-11-03 22:27:48.519634254 +0000 UTC m=+249.141942587" Nov 03 22:27:48.545390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:48.545357 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m45c8"] Nov 03 22:27:48.545608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:48.545582 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-m45c8" podUID="2c63a1e0-f9c6-4010-bdf9-45e1e567492e" containerName="authorino" containerID="cri-o://1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885" gracePeriod=30 Nov 03 22:27:48.772885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:48.772862 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m45c8" Nov 03 22:27:48.946898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:48.946823 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c63a1e0-f9c6-4010-bdf9-45e1e567492e-kube-api-access-z7qmn\" (UniqueName: \"kubernetes.io/projected/2c63a1e0-f9c6-4010-bdf9-45e1e567492e-kube-api-access-z7qmn\") pod \"2c63a1e0-f9c6-4010-bdf9-45e1e567492e\" (UID: \"2c63a1e0-f9c6-4010-bdf9-45e1e567492e\") " Nov 03 22:27:48.949087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:48.949057 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c63a1e0-f9c6-4010-bdf9-45e1e567492e-kube-api-access-z7qmn" pod "2c63a1e0-f9c6-4010-bdf9-45e1e567492e" (UID: "2c63a1e0-f9c6-4010-bdf9-45e1e567492e"). InnerVolumeSpecName "kube-api-access-z7qmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:49.047680 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.047631 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qmn\" (UniqueName: \"kubernetes.io/projected/2c63a1e0-f9c6-4010-bdf9-45e1e567492e-kube-api-access-z7qmn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:49.171101 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.171024 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:27:49.171407 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.171390 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2c63a1e0-f9c6-4010-bdf9-45e1e567492e" containerName="authorino" Nov 03 22:27:49.171499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.171410 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c63a1e0-f9c6-4010-bdf9-45e1e567492e" containerName="authorino" Nov 03 22:27:49.171550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.171519 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c63a1e0-f9c6-4010-bdf9-45e1e567492e" containerName="authorino" Nov 03 22:27:49.178325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.178279 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:49.231915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.229040 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 22:27:49.231915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.230413 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:27:49.350440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.350385 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wlhq\" (UniqueName: \"kubernetes.io/projected/c975eec3-d424-46fc-9518-172b607bd071-kube-api-access-6wlhq\") pod \"limitador-limitador-7c9b87d574-2zdxm\" (UID: \"c975eec3-d424-46fc-9518-172b607bd071\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:49.350440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.350443 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c975eec3-d424-46fc-9518-172b607bd071-config-file\") pod \"limitador-limitador-7c9b87d574-2zdxm\" (UID: \"c975eec3-d424-46fc-9518-172b607bd071\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:49.414839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.413848 5272 generic.go:356] "Generic (PLEG): container finished" podID="2c63a1e0-f9c6-4010-bdf9-45e1e567492e" containerID="1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885" exitCode=0 Nov 03 22:27:49.414839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.414523 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-m45c8" Nov 03 22:27:49.415484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.415452 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m45c8" event={"ID":"2c63a1e0-f9c6-4010-bdf9-45e1e567492e","Type":"ContainerDied","Data":"1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885"} Nov 03 22:27:49.415608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.415501 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-m45c8" event={"ID":"2c63a1e0-f9c6-4010-bdf9-45e1e567492e","Type":"ContainerDied","Data":"6a1baeca426cc1c9195dfcdb851f7c8168d48f9e9bfeefb263256d74ccac5b17"} Nov 03 22:27:49.415608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.415525 5272 scope.go:122] "RemoveContainer" containerID="1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885" Nov 03 22:27:49.427195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.426657 5272 scope.go:122] "RemoveContainer" containerID="1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885" Nov 03 22:27:49.427732 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:27:49.427705 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885\": container with ID starting with 1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885 not found: ID does not exist" containerID="1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885" Nov 03 22:27:49.427887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.427744 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885"} err="failed to get container status \"1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885\": rpc error: code = NotFound desc = could not find container \"1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885\": container with ID starting with 1ab28c92348ecb1d57489ca3262a5c65fc6b0b769fe7df738c50e30a2efcf885 not found: ID does not exist" Nov 03 22:27:49.451740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.451714 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wlhq\" (UniqueName: \"kubernetes.io/projected/c975eec3-d424-46fc-9518-172b607bd071-kube-api-access-6wlhq\") pod \"limitador-limitador-7c9b87d574-2zdxm\" (UID: \"c975eec3-d424-46fc-9518-172b607bd071\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:49.451857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.451766 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c975eec3-d424-46fc-9518-172b607bd071-config-file\") pod \"limitador-limitador-7c9b87d574-2zdxm\" (UID: \"c975eec3-d424-46fc-9518-172b607bd071\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:49.452567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.452543 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c975eec3-d424-46fc-9518-172b607bd071-config-file\") pod \"limitador-limitador-7c9b87d574-2zdxm\" (UID: \"c975eec3-d424-46fc-9518-172b607bd071\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:49.482617 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.482560 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:27:49.522789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.522761 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wlhq\" (UniqueName: \"kubernetes.io/projected/c975eec3-d424-46fc-9518-172b607bd071-kube-api-access-6wlhq\") pod \"limitador-limitador-7c9b87d574-2zdxm\" (UID: \"c975eec3-d424-46fc-9518-172b607bd071\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:49.574420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.574390 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m45c8"] Nov 03 22:27:49.579395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.579367 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-m45c8"] Nov 03 22:27:49.801484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:49.801453 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:50.175826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:50.175731 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:27:50.183563 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:27:50.183534 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc975eec3_d424_46fc_9518_172b607bd071.slice/crio-94afea33f4e18ad534a15e2c43eecf15200d0c32c4e47012c240f1d27a9e0f76 WatchSource:0}: Error finding container 94afea33f4e18ad534a15e2c43eecf15200d0c32c4e47012c240f1d27a9e0f76: Status 404 returned error can't find the container with id 94afea33f4e18ad534a15e2c43eecf15200d0c32c4e47012c240f1d27a9e0f76 Nov 03 22:27:50.419644 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:50.419603 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" event={"ID":"c975eec3-d424-46fc-9518-172b607bd071","Type":"ContainerStarted","Data":"94afea33f4e18ad534a15e2c43eecf15200d0c32c4e47012c240f1d27a9e0f76"} Nov 03 22:27:51.491246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:51.491210 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c63a1e0-f9c6-4010-bdf9-45e1e567492e" path="/var/lib/kubelet/pods/2c63a1e0-f9c6-4010-bdf9-45e1e567492e/volumes" Nov 03 22:27:53.434652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:53.434617 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" event={"ID":"c975eec3-d424-46fc-9518-172b607bd071","Type":"ContainerStarted","Data":"346fe0a19a7dcc9325eda95c8c8486dcb7eab1ff68bccec5bb5a406e22e53950"} Nov 03 22:27:53.435114 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:53.434718 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:27:53.436123 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:53.436102 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" event={"ID":"9c84240c-84ab-43b3-ba71-1235367c0987","Type":"ContainerStarted","Data":"f60e4055b65bcd312e2302cf106a9c22cf006a75c063376e4dc9506cbbbf9e1e"} Nov 03 22:27:53.544403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:53.544359 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" podStartSLOduration=2.303278599 podStartE2EDuration="4.544347941s" totalImagesPullingTime="2.241069342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:53.543482512 +0000 UTC m=+254.165790837" watchObservedRunningTime="2026-11-03 22:27:53.544347941 +0000 UTC m=+254.166656246" Nov 03 22:27:53.652025 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:27:53.651981 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8p92b" podStartSLOduration=2.6072315230000003 podStartE2EDuration="8.651967282s" totalImagesPullingTime="6.044735759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:53.650865824 +0000 UTC m=+254.273174131" watchObservedRunningTime="2026-11-03 22:27:53.651967282 +0000 UTC m=+254.274275589" Nov 03 22:27:54.498927 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:27:54.498895 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:28:04.440605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:04.440574 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2zdxm" Nov 03 22:28:24.498934 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:28:24.498855 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:28:31.045572 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.045536 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6"] Nov 03 22:28:31.067978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.067949 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.151725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.151692 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6"] Nov 03 22:28:31.168757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.168717 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.168982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.168766 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.168982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.168824 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.168982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.168930 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.169167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.169057 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcs8\" (UniqueName: \"kubernetes.io/projected/ed95a603-37ae-43d0-a834-8598e3cd3ceb-kube-api-access-2vcs8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.169167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.169095 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.169276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.169196 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed95a603-37ae-43d0-a834-8598e3cd3ceb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.270449 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.270415 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed95a603-37ae-43d0-a834-8598e3cd3ceb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.270619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.270464 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.270619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.270488 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.270619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.270521 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.270619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.270546 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.270619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.270602 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcs8\" (UniqueName: \"kubernetes.io/projected/ed95a603-37ae-43d0-a834-8598e3cd3ceb-kube-api-access-2vcs8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.270909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.270629 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.271420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.271390 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.272855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.272825 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed95a603-37ae-43d0-a834-8598e3cd3ceb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.273024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.273007 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.273086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.273053 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.273145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.273097 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.334498 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.334436 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed95a603-37ae-43d0-a834-8598e3cd3ceb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.334611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.334516 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcs8\" (UniqueName: \"kubernetes.io/projected/ed95a603-37ae-43d0-a834-8598e3cd3ceb-kube-api-access-2vcs8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8kwp6\" (UID: \"ed95a603-37ae-43d0-a834-8598e3cd3ceb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.378029 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.377993 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:31.798613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.798580 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6"] Nov 03 22:28:31.805418 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:28:31.805394 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded95a603_37ae_43d0_a834_8598e3cd3ceb.slice/crio-5ee25f5e41d22e62c859784f1a6edbd87a8e66d993fc085867959baeb82d1cd7 WatchSource:0}: Error finding container 5ee25f5e41d22e62c859784f1a6edbd87a8e66d993fc085867959baeb82d1cd7: Status 404 returned error can't find the container with id 5ee25f5e41d22e62c859784f1a6edbd87a8e66d993fc085867959baeb82d1cd7 Nov 03 22:28:31.807495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.807463 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:31.807551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:31.807526 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:32.581575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:32.581541 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" event={"ID":"ed95a603-37ae-43d0-a834-8598e3cd3ceb","Type":"ContainerStarted","Data":"f8f518a2c5f40169bea883c21fe66b1d9e0d087da0f987534e23aa2e87511787"} Nov 03 22:28:32.581980 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:32.581590 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" event={"ID":"ed95a603-37ae-43d0-a834-8598e3cd3ceb","Type":"ContainerStarted","Data":"5ee25f5e41d22e62c859784f1a6edbd87a8e66d993fc085867959baeb82d1cd7"} Nov 03 22:28:32.581980 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:32.581661 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:32.583245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:32.583219 5272 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8kwp6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:28:32.583355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:32.583264 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" podUID="ed95a603-37ae-43d0-a834-8598e3cd3ceb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:28:32.697137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:32.697091 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" podStartSLOduration=1.697078485 podStartE2EDuration="1.697078485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:32.696475276 +0000 UTC m=+293.318783582" watchObservedRunningTime="2026-11-03 22:28:32.697078485 +0000 UTC m=+293.319386791" Nov 03 22:28:33.586489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:33.586464 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8kwp6" Nov 03 22:28:33.793877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:33.793845 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j"] Nov 03 22:28:33.794094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:33.794071 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" podUID="75f3c6da-bee7-4227-8140-4a5700282853" containerName="discovery" containerID="cri-o://1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145" gracePeriod=30 Nov 03 22:28:34.027548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.027527 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:28:34.094498 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094476 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-istio-token\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-istio-token\") pod \"75f3c6da-bee7-4227-8140-4a5700282853\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " Nov 03 22:28:34.094620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094522 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-dns-cert\") pod \"75f3c6da-bee7-4227-8140-4a5700282853\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " Nov 03 22:28:34.094620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094590 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-ca-configmap\") pod \"75f3c6da-bee7-4227-8140-4a5700282853\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " Nov 03 22:28:34.094620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094615 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-kube-api-access-tq26v\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-kube-api-access-tq26v\") pod \"75f3c6da-bee7-4227-8140-4a5700282853\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " Nov 03 22:28:34.094774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094661 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-cacerts\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-cacerts\") pod \"75f3c6da-bee7-4227-8140-4a5700282853\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " Nov 03 22:28:34.094774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094705 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-kubeconfig\") pod \"75f3c6da-bee7-4227-8140-4a5700282853\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " Nov 03 22:28:34.094774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094744 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f3c6da-bee7-4227-8140-4a5700282853-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f3c6da-bee7-4227-8140-4a5700282853-local-certs\") pod \"75f3c6da-bee7-4227-8140-4a5700282853\" (UID: \"75f3c6da-bee7-4227-8140-4a5700282853\") " Nov 03 22:28:34.094997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.094963 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-ca-configmap" pod "75f3c6da-bee7-4227-8140-4a5700282853" (UID: "75f3c6da-bee7-4227-8140-4a5700282853"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:34.096908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.096883 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-istio-token" pod "75f3c6da-bee7-4227-8140-4a5700282853" (UID: "75f3c6da-bee7-4227-8140-4a5700282853"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:34.097014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.096913 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-kube-api-access-tq26v" pod "75f3c6da-bee7-4227-8140-4a5700282853" (UID: "75f3c6da-bee7-4227-8140-4a5700282853"). InnerVolumeSpecName "kube-api-access-tq26v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:34.097077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.097052 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-dns-cert" pod "75f3c6da-bee7-4227-8140-4a5700282853" (UID: "75f3c6da-bee7-4227-8140-4a5700282853"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:34.097129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.097075 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-kubeconfig" pod "75f3c6da-bee7-4227-8140-4a5700282853" (UID: "75f3c6da-bee7-4227-8140-4a5700282853"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:34.097129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.097090 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-cacerts" pod "75f3c6da-bee7-4227-8140-4a5700282853" (UID: "75f3c6da-bee7-4227-8140-4a5700282853"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:34.097231 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.097211 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f3c6da-bee7-4227-8140-4a5700282853-local-certs" pod "75f3c6da-bee7-4227-8140-4a5700282853" (UID: "75f3c6da-bee7-4227-8140-4a5700282853"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:28:34.196360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.196222 5272 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-kubeconfig\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:34.196360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.196255 5272 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f3c6da-bee7-4227-8140-4a5700282853-local-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:34.196360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.196272 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:34.196360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.196287 5272 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-dns-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:34.196360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.196302 5272 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75f3c6da-bee7-4227-8140-4a5700282853-istio-csr-ca-configmap\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:34.196360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.196315 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq26v\" (UniqueName: \"kubernetes.io/projected/75f3c6da-bee7-4227-8140-4a5700282853-kube-api-access-tq26v\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:34.196360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.196333 5272 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/75f3c6da-bee7-4227-8140-4a5700282853-cacerts\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:34.591232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.591198 5272 generic.go:356] "Generic (PLEG): container finished" podID="75f3c6da-bee7-4227-8140-4a5700282853" containerID="1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145" exitCode=0 Nov 03 22:28:34.591632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.591250 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" Nov 03 22:28:34.591632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.591279 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" event={"ID":"75f3c6da-bee7-4227-8140-4a5700282853","Type":"ContainerDied","Data":"1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145"} Nov 03 22:28:34.591632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.591327 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j" event={"ID":"75f3c6da-bee7-4227-8140-4a5700282853","Type":"ContainerDied","Data":"ec38d864f9e98f9d3cd9fc00ce1f44a9c024c3efbcd4144b6027146d57076708"} Nov 03 22:28:34.591632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.591351 5272 scope.go:122] "RemoveContainer" containerID="1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145" Nov 03 22:28:34.599471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.599457 5272 scope.go:122] "RemoveContainer" containerID="1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145" Nov 03 22:28:34.599685 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:28:34.599668 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145\": container with ID starting with 1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145 not found: ID does not exist" containerID="1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145" Nov 03 22:28:34.599721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.599694 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145"} err="failed to get container status \"1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145\": rpc error: code = NotFound desc = could not find container \"1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145\": container with ID starting with 1c738c0a19302773e9fc6d7d25016bfc09fb0d290f40994bb13f4cab59d54145 not found: ID does not exist" Nov 03 22:28:34.744275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.744238 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j"] Nov 03 22:28:34.790324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:34.790294 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tbw6j"] Nov 03 22:28:35.490455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:35.490422 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f3c6da-bee7-4227-8140-4a5700282853" path="/var/lib/kubelet/pods/75f3c6da-bee7-4227-8140-4a5700282853/volumes" Nov 03 22:28:37.139929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.139895 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-82qwd"] Nov 03 22:28:37.140408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.140179 5272 cpu_manager.go:414] "containerMap: removing container" podUID="75f3c6da-bee7-4227-8140-4a5700282853" containerName="discovery" Nov 03 22:28:37.140408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.140190 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f3c6da-bee7-4227-8140-4a5700282853" containerName="discovery" Nov 03 22:28:37.140408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.140238 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f3c6da-bee7-4227-8140-4a5700282853" containerName="discovery" Nov 03 22:28:37.144400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.144381 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.220667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.220631 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwd5\" (UniqueName: \"kubernetes.io/projected/14e070a9-8f8f-413b-9e84-bb8c44d4feca-kube-api-access-lmwd5\") pod \"model-serving-api-75bd89fdfc-82qwd\" (UID: \"14e070a9-8f8f-413b-9e84-bb8c44d4feca\") " pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.220854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.220686 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e070a9-8f8f-413b-9e84-bb8c44d4feca-tls-certs\") pod \"model-serving-api-75bd89fdfc-82qwd\" (UID: \"14e070a9-8f8f-413b-9e84-bb8c44d4feca\") " pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.241630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.241597 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:28:37.241630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.241618 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:28:37.241792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.241638 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-86l6m\"" Nov 03 22:28:37.241792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.241618 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 22:28:37.242627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.242609 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-82qwd"] Nov 03 22:28:37.321835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.321790 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwd5\" (UniqueName: \"kubernetes.io/projected/14e070a9-8f8f-413b-9e84-bb8c44d4feca-kube-api-access-lmwd5\") pod \"model-serving-api-75bd89fdfc-82qwd\" (UID: \"14e070a9-8f8f-413b-9e84-bb8c44d4feca\") " pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.322002 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.321870 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e070a9-8f8f-413b-9e84-bb8c44d4feca-tls-certs\") pod \"model-serving-api-75bd89fdfc-82qwd\" (UID: \"14e070a9-8f8f-413b-9e84-bb8c44d4feca\") " pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.324176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.324156 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e070a9-8f8f-413b-9e84-bb8c44d4feca-tls-certs\") pod \"model-serving-api-75bd89fdfc-82qwd\" (UID: \"14e070a9-8f8f-413b-9e84-bb8c44d4feca\") " pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.379573 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.379536 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwd5\" (UniqueName: \"kubernetes.io/projected/14e070a9-8f8f-413b-9e84-bb8c44d4feca-kube-api-access-lmwd5\") pod \"model-serving-api-75bd89fdfc-82qwd\" (UID: \"14e070a9-8f8f-413b-9e84-bb8c44d4feca\") " pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.456695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.456611 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:37.826960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:37.826926 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-82qwd"] Nov 03 22:28:37.834784 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:28:37.834754 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e070a9_8f8f_413b_9e84_bb8c44d4feca.slice/crio-0217eff3669c7ba5ed71910fc188ebcd396c9957c4ffc251301290a9ca9276b2 WatchSource:0}: Error finding container 0217eff3669c7ba5ed71910fc188ebcd396c9957c4ffc251301290a9ca9276b2: Status 404 returned error can't find the container with id 0217eff3669c7ba5ed71910fc188ebcd396c9957c4ffc251301290a9ca9276b2 Nov 03 22:28:38.608734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:38.608698 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" event={"ID":"14e070a9-8f8f-413b-9e84-bb8c44d4feca","Type":"ContainerStarted","Data":"0217eff3669c7ba5ed71910fc188ebcd396c9957c4ffc251301290a9ca9276b2"} Nov 03 22:28:40.618364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:40.618323 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" event={"ID":"14e070a9-8f8f-413b-9e84-bb8c44d4feca","Type":"ContainerStarted","Data":"3063d0312f99c214aa3d23cddf71be038c10694df9b66e57ea05ff726404c9eb"} Nov 03 22:28:40.618795 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:40.618413 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:40.727750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:40.727652 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" podStartSLOduration=1.171936206 podStartE2EDuration="3.727638144s" totalImagesPullingTime="2.555701938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:40.727299512 +0000 UTC m=+301.349607820" watchObservedRunningTime="2026-11-03 22:28:40.727638144 +0000 UTC m=+301.349946460" Nov 03 22:28:51.626867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:28:51.626843 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-82qwd" Nov 03 22:28:54.497900 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:28:54.497866 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:29:20.470947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:20.470914 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:29:24.499090 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:29:24.499046 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:29:24.709267 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:24.709232 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-m99tr"] Nov 03 22:29:24.712499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:24.712478 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" Nov 03 22:29:24.765429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:24.765370 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-m99tr"] Nov 03 22:29:24.772641 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:24.772614 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48cm\" (UniqueName: \"kubernetes.io/projected/75f336b5-f452-426d-8059-dc377fddeb31-kube-api-access-h48cm\") pod \"authorino-6f6944bcdd-m99tr\" (UID: \"75f336b5-f452-426d-8059-dc377fddeb31\") " pod="kuadrant-system/authorino-6f6944bcdd-m99tr" Nov 03 22:29:24.873470 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:24.873441 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48cm\" (UniqueName: \"kubernetes.io/projected/75f336b5-f452-426d-8059-dc377fddeb31-kube-api-access-h48cm\") pod \"authorino-6f6944bcdd-m99tr\" (UID: \"75f336b5-f452-426d-8059-dc377fddeb31\") " pod="kuadrant-system/authorino-6f6944bcdd-m99tr" Nov 03 22:29:24.932103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:24.932080 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48cm\" (UniqueName: \"kubernetes.io/projected/75f336b5-f452-426d-8059-dc377fddeb31-kube-api-access-h48cm\") pod \"authorino-6f6944bcdd-m99tr\" (UID: \"75f336b5-f452-426d-8059-dc377fddeb31\") " pod="kuadrant-system/authorino-6f6944bcdd-m99tr" Nov 03 22:29:25.022931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.022878 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" Nov 03 22:29:25.086982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.086953 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-m99tr"] Nov 03 22:29:25.111607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.111573 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86648c9d7b-cg5pf"] Nov 03 22:29:25.115119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.115101 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" Nov 03 22:29:25.167505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.167471 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86648c9d7b-cg5pf"] Nov 03 22:29:25.175972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.175952 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqf5x\" (UniqueName: \"kubernetes.io/projected/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2-kube-api-access-tqf5x\") pod \"authorino-86648c9d7b-cg5pf\" (UID: \"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2\") " pod="kuadrant-system/authorino-86648c9d7b-cg5pf" Nov 03 22:29:25.277110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.277056 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqf5x\" (UniqueName: \"kubernetes.io/projected/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2-kube-api-access-tqf5x\") pod \"authorino-86648c9d7b-cg5pf\" (UID: \"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2\") " pod="kuadrant-system/authorino-86648c9d7b-cg5pf" Nov 03 22:29:25.334067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.334044 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqf5x\" (UniqueName: \"kubernetes.io/projected/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2-kube-api-access-tqf5x\") pod \"authorino-86648c9d7b-cg5pf\" (UID: \"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2\") " pod="kuadrant-system/authorino-86648c9d7b-cg5pf" Nov 03 22:29:25.388793 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.388766 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-m99tr"] Nov 03 22:29:25.395989 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:29:25.395965 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f336b5_f452_426d_8059_dc377fddeb31.slice/crio-f6ad7bdd6645e7ee3bc2c772d11184987e2e01f654d602d64ae1c2465f549bff WatchSource:0}: Error finding container f6ad7bdd6645e7ee3bc2c772d11184987e2e01f654d602d64ae1c2465f549bff: Status 404 returned error can't find the container with id f6ad7bdd6645e7ee3bc2c772d11184987e2e01f654d602d64ae1c2465f549bff Nov 03 22:29:25.426659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.426640 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" Nov 03 22:29:25.559973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.559940 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86648c9d7b-cg5pf"] Nov 03 22:29:25.588410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.588383 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68cc986d58-f747q"] Nov 03 22:29:25.591700 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.591681 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:25.640491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.640472 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 22:29:25.644975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.644951 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68cc986d58-f747q"] Nov 03 22:29:25.681491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.681465 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-tls-cert\") pod \"authorino-68cc986d58-f747q\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:25.681596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.681498 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nzgs\" (UniqueName: \"kubernetes.io/projected/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-kube-api-access-5nzgs\") pod \"authorino-68cc986d58-f747q\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:25.781987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.781962 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-tls-cert\") pod \"authorino-68cc986d58-f747q\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:25.782081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.781990 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nzgs\" (UniqueName: \"kubernetes.io/projected/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-kube-api-access-5nzgs\") pod \"authorino-68cc986d58-f747q\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:25.784254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.784235 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-tls-cert\") pod \"authorino-68cc986d58-f747q\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:25.789469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.789450 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86648c9d7b-cg5pf"] Nov 03 22:29:25.793214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.793185 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" event={"ID":"75f336b5-f452-426d-8059-dc377fddeb31","Type":"ContainerStarted","Data":"f6ad7bdd6645e7ee3bc2c772d11184987e2e01f654d602d64ae1c2465f549bff"} Nov 03 22:29:25.796881 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:29:25.796856 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabe05fd5_ce4d_4b0b_aa2e_4c04c2087af2.slice/crio-ebfaa1894c44aae461cbdded81c93170b2a2f005e1f090d2cc510117abebcba6 WatchSource:0}: Error finding container ebfaa1894c44aae461cbdded81c93170b2a2f005e1f090d2cc510117abebcba6: Status 404 returned error can't find the container with id ebfaa1894c44aae461cbdded81c93170b2a2f005e1f090d2cc510117abebcba6 Nov 03 22:29:25.839217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.839171 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nzgs\" (UniqueName: \"kubernetes.io/projected/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-kube-api-access-5nzgs\") pod \"authorino-68cc986d58-f747q\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:25.902686 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:25.902669 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:29:26.267833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:26.267747 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68cc986d58-f747q"] Nov 03 22:29:26.275081 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:29:26.275059 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ac61a7e_338b_4f0e_8950_8a4cf4eb1692.slice/crio-4b99041c429384e83bcfeb4cf0297979019478838308bbb98423bae33e1d2c00 WatchSource:0}: Error finding container 4b99041c429384e83bcfeb4cf0297979019478838308bbb98423bae33e1d2c00: Status 404 returned error can't find the container with id 4b99041c429384e83bcfeb4cf0297979019478838308bbb98423bae33e1d2c00 Nov 03 22:29:26.799198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:26.799157 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" event={"ID":"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2","Type":"ContainerStarted","Data":"ebfaa1894c44aae461cbdded81c93170b2a2f005e1f090d2cc510117abebcba6"} Nov 03 22:29:26.800247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:26.800220 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cc986d58-f747q" event={"ID":"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692","Type":"ContainerStarted","Data":"4b99041c429384e83bcfeb4cf0297979019478838308bbb98423bae33e1d2c00"} Nov 03 22:29:27.806247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:27.806158 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" event={"ID":"75f336b5-f452-426d-8059-dc377fddeb31","Type":"ContainerStarted","Data":"3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b"} Nov 03 22:29:27.806247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:27.806185 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" podUID="75f336b5-f452-426d-8059-dc377fddeb31" containerName="authorino" containerID="cri-o://3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b" gracePeriod=30 Nov 03 22:29:27.807633 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:27.807608 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" event={"ID":"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2","Type":"ContainerStarted","Data":"b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5"} Nov 03 22:29:27.807766 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:27.807653 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" podUID="abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2" containerName="authorino" containerID="cri-o://b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5" gracePeriod=30 Nov 03 22:29:27.809644 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:27.809620 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cc986d58-f747q" event={"ID":"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692","Type":"ContainerStarted","Data":"ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939"} Nov 03 22:29:27.914447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:27.914386 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" podStartSLOduration=2.279348952 podStartE2EDuration="3.914371495s" totalImagesPullingTime="1.635022543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:27.913736282 +0000 UTC m=+348.536044593" watchObservedRunningTime="2026-11-03 22:29:27.914371495 +0000 UTC m=+348.536679796" Nov 03 22:29:28.023340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.023277 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68cc986d58-f747q" podStartSLOduration=1.746136569 podStartE2EDuration="3.023256463s" totalImagesPullingTime="1.277119894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:28.021397791 +0000 UTC m=+348.643706094" watchObservedRunningTime="2026-11-03 22:29:28.023256463 +0000 UTC m=+348.645564771" Nov 03 22:29:28.041336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.041301 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" Nov 03 22:29:28.059685 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.059580 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ql7vl"] Nov 03 22:29:28.059904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.059855 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" podUID="1d7be776-2c84-4863-872f-c511231562e3" containerName="authorino" containerID="cri-o://3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd" gracePeriod=30 Nov 03 22:29:28.100497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.100172 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2-kube-api-access-tqf5x\" (UniqueName: \"kubernetes.io/projected/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2-kube-api-access-tqf5x\") pod \"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2\" (UID: \"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2\") " Nov 03 22:29:28.102926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.102880 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2-kube-api-access-tqf5x" pod "abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2" (UID: "abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2"). InnerVolumeSpecName "kube-api-access-tqf5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:28.130938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.130910 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" Nov 03 22:29:28.201272 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.201230 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f336b5-f452-426d-8059-dc377fddeb31-kube-api-access-h48cm\" (UniqueName: \"kubernetes.io/projected/75f336b5-f452-426d-8059-dc377fddeb31-kube-api-access-h48cm\") pod \"75f336b5-f452-426d-8059-dc377fddeb31\" (UID: \"75f336b5-f452-426d-8059-dc377fddeb31\") " Nov 03 22:29:28.201455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.201366 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqf5x\" (UniqueName: \"kubernetes.io/projected/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2-kube-api-access-tqf5x\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:28.203192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.203160 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f336b5-f452-426d-8059-dc377fddeb31-kube-api-access-h48cm" pod "75f336b5-f452-426d-8059-dc377fddeb31" (UID: "75f336b5-f452-426d-8059-dc377fddeb31"). InnerVolumeSpecName "kube-api-access-h48cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:28.235566 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.235427 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" podStartSLOduration=1.794989194 podStartE2EDuration="3.235407281s" totalImagesPullingTime="1.440418087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:28.128292605 +0000 UTC m=+348.750600910" watchObservedRunningTime="2026-11-03 22:29:28.235407281 +0000 UTC m=+348.857715588" Nov 03 22:29:28.302381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.302356 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" Nov 03 22:29:28.302546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.302522 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48cm\" (UniqueName: \"kubernetes.io/projected/75f336b5-f452-426d-8059-dc377fddeb31-kube-api-access-h48cm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:28.403010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.402920 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7be776-2c84-4863-872f-c511231562e3-kube-api-access-bbxz7\" (UniqueName: \"kubernetes.io/projected/1d7be776-2c84-4863-872f-c511231562e3-kube-api-access-bbxz7\") pod \"1d7be776-2c84-4863-872f-c511231562e3\" (UID: \"1d7be776-2c84-4863-872f-c511231562e3\") " Nov 03 22:29:28.405468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.405428 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7be776-2c84-4863-872f-c511231562e3-kube-api-access-bbxz7" pod "1d7be776-2c84-4863-872f-c511231562e3" (UID: "1d7be776-2c84-4863-872f-c511231562e3"). InnerVolumeSpecName "kube-api-access-bbxz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:28.503933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.503892 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbxz7\" (UniqueName: \"kubernetes.io/projected/1d7be776-2c84-4863-872f-c511231562e3-kube-api-access-bbxz7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:28.815084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.815046 5272 generic.go:356] "Generic (PLEG): container finished" podID="abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2" containerID="b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5" exitCode=0 Nov 03 22:29:28.815499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.815091 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" Nov 03 22:29:28.815499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.815140 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" event={"ID":"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2","Type":"ContainerDied","Data":"b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5"} Nov 03 22:29:28.815499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.815183 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86648c9d7b-cg5pf" event={"ID":"abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2","Type":"ContainerDied","Data":"ebfaa1894c44aae461cbdded81c93170b2a2f005e1f090d2cc510117abebcba6"} Nov 03 22:29:28.815499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.815199 5272 scope.go:122] "RemoveContainer" containerID="b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5" Nov 03 22:29:28.816443 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.816416 5272 generic.go:356] "Generic (PLEG): container finished" podID="1d7be776-2c84-4863-872f-c511231562e3" containerID="3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd" exitCode=0 Nov 03 22:29:28.816548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.816449 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" event={"ID":"1d7be776-2c84-4863-872f-c511231562e3","Type":"ContainerDied","Data":"3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd"} Nov 03 22:29:28.816548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.816470 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" Nov 03 22:29:28.816548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.816500 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ql7vl" event={"ID":"1d7be776-2c84-4863-872f-c511231562e3","Type":"ContainerDied","Data":"def5b16b5962678ad949933ed2dccf7ee307603f17392f36cd61a2fd0535f1ed"} Nov 03 22:29:28.817883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.817860 5272 generic.go:356] "Generic (PLEG): container finished" podID="75f336b5-f452-426d-8059-dc377fddeb31" containerID="3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b" exitCode=0 Nov 03 22:29:28.817976 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.817901 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" Nov 03 22:29:28.817976 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.817936 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" event={"ID":"75f336b5-f452-426d-8059-dc377fddeb31","Type":"ContainerDied","Data":"3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b"} Nov 03 22:29:28.818051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.817966 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-m99tr" event={"ID":"75f336b5-f452-426d-8059-dc377fddeb31","Type":"ContainerDied","Data":"f6ad7bdd6645e7ee3bc2c772d11184987e2e01f654d602d64ae1c2465f549bff"} Nov 03 22:29:28.825023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.825007 5272 scope.go:122] "RemoveContainer" containerID="b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5" Nov 03 22:29:28.825305 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:29:28.825282 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5\": container with ID starting with b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5 not found: ID does not exist" containerID="b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5" Nov 03 22:29:28.825375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.825312 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5"} err="failed to get container status \"b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5\": rpc error: code = NotFound desc = could not find container \"b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5\": container with ID starting with b93da073395ad2479435fc4d03c8c2a52a797e2996535908ad1b8229cc3714f5 not found: ID does not exist" Nov 03 22:29:28.825375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.825329 5272 scope.go:122] "RemoveContainer" containerID="3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd" Nov 03 22:29:28.832676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.832651 5272 scope.go:122] "RemoveContainer" containerID="3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd" Nov 03 22:29:28.832954 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:29:28.832934 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd\": container with ID starting with 3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd not found: ID does not exist" containerID="3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd" Nov 03 22:29:28.833007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.832962 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd"} err="failed to get container status \"3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd\": rpc error: code = NotFound desc = could not find container \"3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd\": container with ID starting with 3178b771311f017263ce0e0852551c2908f0cac32d1eecc960da84d52192d8dd not found: ID does not exist" Nov 03 22:29:28.833007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.832977 5272 scope.go:122] "RemoveContainer" containerID="3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b" Nov 03 22:29:28.839930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.839915 5272 scope.go:122] "RemoveContainer" containerID="3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b" Nov 03 22:29:28.840184 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:29:28.840166 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b\": container with ID starting with 3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b not found: ID does not exist" containerID="3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b" Nov 03 22:29:28.840239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.840188 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b"} err="failed to get container status \"3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b\": rpc error: code = NotFound desc = could not find container \"3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b\": container with ID starting with 3d35cbe35274354f340fe08d7bf13b3930758b820ee1897ba509ddc2b65ad02b not found: ID does not exist" Nov 03 22:29:28.975701 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.975670 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-m99tr"] Nov 03 22:29:28.979203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:28.979178 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-m99tr"] Nov 03 22:29:29.081624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:29.081536 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86648c9d7b-cg5pf"] Nov 03 22:29:29.082964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:29.082937 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-86648c9d7b-cg5pf"] Nov 03 22:29:29.191981 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:29.191945 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ql7vl"] Nov 03 22:29:29.210920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:29.210889 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ql7vl"] Nov 03 22:29:29.490589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:29.490514 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7be776-2c84-4863-872f-c511231562e3" path="/var/lib/kubelet/pods/1d7be776-2c84-4863-872f-c511231562e3/volumes" Nov 03 22:29:29.490837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:29.490823 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f336b5-f452-426d-8059-dc377fddeb31" path="/var/lib/kubelet/pods/75f336b5-f452-426d-8059-dc377fddeb31/volumes" Nov 03 22:29:29.491095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:29:29.491084 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2" path="/var/lib/kubelet/pods/abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2/volumes" Nov 03 22:29:54.499393 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:29:54.499308 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:30:00.176608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.176577 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc"] Nov 03 22:30:00.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.176913 5272 cpu_manager.go:414] "containerMap: removing container" podUID="abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2" containerName="authorino" Nov 03 22:30:00.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.176924 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2" containerName="authorino" Nov 03 22:30:00.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.176937 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1d7be776-2c84-4863-872f-c511231562e3" containerName="authorino" Nov 03 22:30:00.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.176942 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7be776-2c84-4863-872f-c511231562e3" containerName="authorino" Nov 03 22:30:00.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.176957 5272 cpu_manager.go:414] "containerMap: removing container" podUID="75f336b5-f452-426d-8059-dc377fddeb31" containerName="authorino" Nov 03 22:30:00.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.176962 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f336b5-f452-426d-8059-dc377fddeb31" containerName="authorino" Nov 03 22:30:00.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.177005 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe05fd5-ce4d-4b0b-aa2e-4c04c2087af2" containerName="authorino" Nov 03 22:30:00.177221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.177013 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7be776-2c84-4863-872f-c511231562e3" containerName="authorino" Nov 03 22:30:00.177221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.177020 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f336b5-f452-426d-8059-dc377fddeb31" containerName="authorino" Nov 03 22:30:00.179787 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.179769 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.228953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.228922 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:30:00.252262 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.252231 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwg5\" (UniqueName: \"kubernetes.io/projected/f9c56569-c593-4297-8b71-47b0fb6589cd-kube-api-access-ldwg5\") pod \"maas-api-key-cleanup-29895750-gbvxc\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.252418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.252299 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c56569-c593-4297-8b71-47b0fb6589cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-gbvxc\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.275308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.275278 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:30:00.275427 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.275323 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:30:00.276238 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.276215 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc"] Nov 03 22:30:00.353055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.353022 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwg5\" (UniqueName: \"kubernetes.io/projected/f9c56569-c593-4297-8b71-47b0fb6589cd-kube-api-access-ldwg5\") pod \"maas-api-key-cleanup-29895750-gbvxc\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.353219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.353104 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c56569-c593-4297-8b71-47b0fb6589cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-gbvxc\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.353713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.353688 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c56569-c593-4297-8b71-47b0fb6589cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-gbvxc\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.410518 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.410490 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwg5\" (UniqueName: \"kubernetes.io/projected/f9c56569-c593-4297-8b71-47b0fb6589cd-kube-api-access-ldwg5\") pod \"maas-api-key-cleanup-29895750-gbvxc\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.491328 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.491250 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:00.856198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.856162 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc"] Nov 03 22:30:00.863848 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:30:00.863816 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c56569_c593_4297_8b71_47b0fb6589cd.slice/crio-e256f1fb891b2453a31d93039d7ea537114c699ab1356113b2f6b6e23aa5e938 WatchSource:0}: Error finding container e256f1fb891b2453a31d93039d7ea537114c699ab1356113b2f6b6e23aa5e938: Status 404 returned error can't find the container with id e256f1fb891b2453a31d93039d7ea537114c699ab1356113b2f6b6e23aa5e938 Nov 03 22:30:00.944920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:00.944886 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" event={"ID":"f9c56569-c593-4297-8b71-47b0fb6589cd","Type":"ContainerStarted","Data":"e256f1fb891b2453a31d93039d7ea537114c699ab1356113b2f6b6e23aa5e938"} Nov 03 22:30:06.972187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:06.972107 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" event={"ID":"f9c56569-c593-4297-8b71-47b0fb6589cd","Type":"ContainerStarted","Data":"441cd82d92900698ec047d01982f74d6fe8c4c0cf99bfa1edca721e49515b18e"} Nov 03 22:30:07.084625 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:07.084449 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" podStartSLOduration=1.414714453 podStartE2EDuration="7.084432805s" totalImagesPullingTime="5.669718352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:07.083967677 +0000 UTC m=+387.706275985" watchObservedRunningTime="2026-11-03 22:30:07.084432805 +0000 UTC m=+387.706741114" Nov 03 22:30:07.980119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:07.980082 5272 generic.go:356] "Generic (PLEG): container finished" podID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerID="441cd82d92900698ec047d01982f74d6fe8c4c0cf99bfa1edca721e49515b18e" exitCode=60 Nov 03 22:30:07.980528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:07.980166 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" event={"ID":"f9c56569-c593-4297-8b71-47b0fb6589cd","Type":"ContainerDied","Data":"441cd82d92900698ec047d01982f74d6fe8c4c0cf99bfa1edca721e49515b18e"} Nov 03 22:30:07.980528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:07.980312 5272 scope.go:122] "RemoveContainer" containerID="441cd82d92900698ec047d01982f74d6fe8c4c0cf99bfa1edca721e49515b18e" Nov 03 22:30:08.985322 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:08.985249 5272 generic.go:356] "Generic (PLEG): container finished" podID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerID="5802a15b4309c34575e69bc842ea1355574ac3b4bdc8c2e72bbfb8230db5cf5a" exitCode=60 Nov 03 22:30:08.985742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:08.985331 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" event={"ID":"f9c56569-c593-4297-8b71-47b0fb6589cd","Type":"ContainerDied","Data":"5802a15b4309c34575e69bc842ea1355574ac3b4bdc8c2e72bbfb8230db5cf5a"} Nov 03 22:30:08.985742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:08.985388 5272 scope.go:122] "RemoveContainer" containerID="441cd82d92900698ec047d01982f74d6fe8c4c0cf99bfa1edca721e49515b18e" Nov 03 22:30:08.985742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:08.985571 5272 scope.go:122] "RemoveContainer" containerID="5802a15b4309c34575e69bc842ea1355574ac3b4bdc8c2e72bbfb8230db5cf5a" Nov 03 22:30:08.985896 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:30:08.985775 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-gbvxc_odh-ai-gateway-infra(f9c56569-c593-4297-8b71-47b0fb6589cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" Nov 03 22:30:09.991051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:09.991025 5272 scope.go:122] "RemoveContainer" containerID="5802a15b4309c34575e69bc842ea1355574ac3b4bdc8c2e72bbfb8230db5cf5a" Nov 03 22:30:09.991427 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:30:09.991217 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-gbvxc_odh-ai-gateway-infra(f9c56569-c593-4297-8b71-47b0fb6589cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" Nov 03 22:30:20.486364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:20.486333 5272 scope.go:122] "RemoveContainer" containerID="5802a15b4309c34575e69bc842ea1355574ac3b4bdc8c2e72bbfb8230db5cf5a" Nov 03 22:30:21.035732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:21.035696 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" event={"ID":"f9c56569-c593-4297-8b71-47b0fb6589cd","Type":"ContainerStarted","Data":"a37cbf5a4d6cd8281a254b4e143ec0ddad912f0d6b978cbe64e7b3546c403490"} Nov 03 22:30:22.042015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:22.041981 5272 generic.go:356] "Generic (PLEG): container finished" podID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerID="a37cbf5a4d6cd8281a254b4e143ec0ddad912f0d6b978cbe64e7b3546c403490" exitCode=60 Nov 03 22:30:22.042392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:22.042022 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" event={"ID":"f9c56569-c593-4297-8b71-47b0fb6589cd","Type":"ContainerDied","Data":"a37cbf5a4d6cd8281a254b4e143ec0ddad912f0d6b978cbe64e7b3546c403490"} Nov 03 22:30:22.042392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:22.042068 5272 scope.go:122] "RemoveContainer" containerID="5802a15b4309c34575e69bc842ea1355574ac3b4bdc8c2e72bbfb8230db5cf5a" Nov 03 22:30:22.042392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:22.042349 5272 scope.go:122] "RemoveContainer" containerID="a37cbf5a4d6cd8281a254b4e143ec0ddad912f0d6b978cbe64e7b3546c403490" Nov 03 22:30:22.042566 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:30:22.042547 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-gbvxc_odh-ai-gateway-infra(f9c56569-c593-4297-8b71-47b0fb6589cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" Nov 03 22:30:22.155085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:22.155056 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc"] Nov 03 22:30:23.167024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:23.166999 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:23.322423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:23.322348 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c56569-c593-4297-8b71-47b0fb6589cd-kube-api-access-ldwg5\" (UniqueName: \"kubernetes.io/projected/f9c56569-c593-4297-8b71-47b0fb6589cd-kube-api-access-ldwg5\") pod \"f9c56569-c593-4297-8b71-47b0fb6589cd\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " Nov 03 22:30:23.322556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:23.322448 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c56569-c593-4297-8b71-47b0fb6589cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c56569-c593-4297-8b71-47b0fb6589cd-maas-api-service-ca\") pod \"f9c56569-c593-4297-8b71-47b0fb6589cd\" (UID: \"f9c56569-c593-4297-8b71-47b0fb6589cd\") " Nov 03 22:30:23.322781 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:23.322757 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c56569-c593-4297-8b71-47b0fb6589cd-maas-api-service-ca" pod "f9c56569-c593-4297-8b71-47b0fb6589cd" (UID: "f9c56569-c593-4297-8b71-47b0fb6589cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:23.324476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:23.324449 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c56569-c593-4297-8b71-47b0fb6589cd-kube-api-access-ldwg5" pod "f9c56569-c593-4297-8b71-47b0fb6589cd" (UID: "f9c56569-c593-4297-8b71-47b0fb6589cd"). InnerVolumeSpecName "kube-api-access-ldwg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:23.423348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:23.423319 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c56569-c593-4297-8b71-47b0fb6589cd-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:23.423348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:23.423343 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwg5\" (UniqueName: \"kubernetes.io/projected/f9c56569-c593-4297-8b71-47b0fb6589cd-kube-api-access-ldwg5\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:24.052009 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:24.051972 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" event={"ID":"f9c56569-c593-4297-8b71-47b0fb6589cd","Type":"ContainerDied","Data":"e256f1fb891b2453a31d93039d7ea537114c699ab1356113b2f6b6e23aa5e938"} Nov 03 22:30:24.052182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:24.052033 5272 scope.go:122] "RemoveContainer" containerID="a37cbf5a4d6cd8281a254b4e143ec0ddad912f0d6b978cbe64e7b3546c403490" Nov 03 22:30:24.052182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:24.051990 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc" Nov 03 22:30:24.204331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:24.204298 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc"] Nov 03 22:30:24.250644 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:24.250609 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gbvxc"] Nov 03 22:30:24.497939 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:30:24.497861 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:30:25.490944 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:25.490912 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" path="/var/lib/kubelet/pods/f9c56569-c593-4297-8b71-47b0fb6589cd/volumes" Nov 03 22:30:42.867550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.867512 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw"] Nov 03 22:30:42.867986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.867958 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.867986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.867978 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.867986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.867988 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.868093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.867993 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.868093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.868002 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.868093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.868006 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.868093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.868065 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.868093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.868073 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:42.875619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.875597 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:42.926797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.926760 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw"] Nov 03 22:30:42.943918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.943894 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 22:30:42.968361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.968330 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:42.968482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.968373 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnn47\" (UniqueName: \"kubernetes.io/projected/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-kube-api-access-dnn47\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:42.968482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.968392 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:42.968482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.968439 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:42.968588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.968524 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:42.968588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.968552 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:42.971926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.971910 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 22:30:42.972117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.972099 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 22:30:42.972164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:42.972116 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pssl4\"" Nov 03 22:30:43.069068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069038 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069075 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnn47\" (UniqueName: \"kubernetes.io/projected/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-kube-api-access-dnn47\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069094 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069110 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069150 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069177 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069533 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069555 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.069711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.069577 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.071344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.071311 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.071461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.071445 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.126634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.126576 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnn47\" (UniqueName: \"kubernetes.io/projected/ca3f872d-bbbb-43e8-bed7-5e6c065a625f-kube-api-access-dnn47\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw\" (UID: \"ca3f872d-bbbb-43e8-bed7-5e6c065a625f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.187928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.187906 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:43.559996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:43.559969 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw"] Nov 03 22:30:43.567098 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:30:43.567073 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3f872d_bbbb_43e8_bed7_5e6c065a625f.slice/crio-12c924feba71004ee2569bdde638d787f0f82b871ff97dbc7570ec65ecd9b5cf WatchSource:0}: Error finding container 12c924feba71004ee2569bdde638d787f0f82b871ff97dbc7570ec65ecd9b5cf: Status 404 returned error can't find the container with id 12c924feba71004ee2569bdde638d787f0f82b871ff97dbc7570ec65ecd9b5cf Nov 03 22:30:44.136021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:44.135990 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" event={"ID":"ca3f872d-bbbb-43e8-bed7-5e6c065a625f","Type":"ContainerStarted","Data":"12c924feba71004ee2569bdde638d787f0f82b871ff97dbc7570ec65ecd9b5cf"} Nov 03 22:30:46.119460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.119428 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ggsmt"] Nov 03 22:30:46.119913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.119788 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c56569-c593-4297-8b71-47b0fb6589cd" containerName="cleanup" Nov 03 22:30:46.122529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.122505 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.193310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.193286 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7bw\" (UniqueName: \"kubernetes.io/projected/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-kube-api-access-rt7bw\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.193422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.193317 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.193422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.193334 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.193540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.193406 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-home\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.193540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.193462 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.193540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.193489 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.216411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.216395 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 22:30:46.218954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.218936 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ggsmt"] Nov 03 22:30:46.294897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.294859 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.294921 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-home\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.294953 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.294976 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.295057 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7bw\" (UniqueName: \"kubernetes.io/projected/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-kube-api-access-rt7bw\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.295083 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.295395 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.295415 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.295673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.295647 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-home\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.297070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.297052 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.297364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.297349 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.352431 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.352391 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7bw\" (UniqueName: \"kubernetes.io/projected/7f7a50d4-86b8-404f-99db-2b3be5d3c7cf-kube-api-access-rt7bw\") pod \"echo-simulator-kserve-64b6f8f96c-ggsmt\" (UID: \"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.435068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.434984 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:46.522185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.522155 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:30:46.806013 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:46.805972 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ggsmt"] Nov 03 22:30:47.149165 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:47.149050 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" event={"ID":"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf","Type":"ContainerStarted","Data":"8276907921c17012efe1c244ff935e4c36836a5e2c7e6b679188f9d2ee080ab8"} Nov 03 22:30:47.150536 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:47.150511 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" event={"ID":"ca3f872d-bbbb-43e8-bed7-5e6c065a625f","Type":"ContainerStarted","Data":"b11c7a914d79ac4db95db57b45ec65bf35a933f289a31d029bda72aba4d18aff"} Nov 03 22:30:47.188495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:47.188471 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:47.261595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:47.261533 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" podStartSLOduration=2.660499259 podStartE2EDuration="5.261513367s" totalImagesPullingTime="2.601014108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:47.260719856 +0000 UTC m=+427.883028163" watchObservedRunningTime="2026-11-03 22:30:47.261513367 +0000 UTC m=+427.883821676" Nov 03 22:30:50.171645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:50.171606 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" event={"ID":"7f7a50d4-86b8-404f-99db-2b3be5d3c7cf","Type":"ContainerStarted","Data":"fd6856993a2a87ad24f8fca5d29dbc446d6cb92ea2a2a481679c9798916bfdb4"} Nov 03 22:30:50.281736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:50.281683 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" podStartSLOduration=1.625460213 podStartE2EDuration="4.281669016s" totalImagesPullingTime="2.656208803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:50.280995294 +0000 UTC m=+430.903303599" watchObservedRunningTime="2026-11-03 22:30:50.281669016 +0000 UTC m=+430.903977302" Nov 03 22:30:50.435665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:50.435596 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:53.188556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:53.188522 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:53.200895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:53.200867 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:54.198266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:54.198241 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw" Nov 03 22:30:54.498976 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:30:54.498904 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:30:55.042643 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.042611 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn"] Nov 03 22:30:55.045783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.045765 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.068294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.068270 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.068375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.068310 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.068375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.068334 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.068542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.068518 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.068586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.068567 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5546\" (UniqueName: \"kubernetes.io/projected/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-kube-api-access-h5546\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.068682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.068665 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.113604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.113580 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 22:30:55.142250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.142229 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn"] Nov 03 22:30:55.169288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169264 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169294 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5546\" (UniqueName: \"kubernetes.io/projected/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-kube-api-access-h5546\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169315 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169340 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169363 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169385 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169689 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169713 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.169818 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.169787 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.171558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.171537 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.171795 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.171781 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.226769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.226744 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5546\" (UniqueName: \"kubernetes.io/projected/1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d-kube-api-access-h5546\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn\" (UID: \"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.358083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.358003 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:55.725791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.725700 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn"] Nov 03 22:30:55.733003 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:30:55.732976 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e27d43a_4ac0_4f7d_87ab_3f74acbbda2d.slice/crio-fe032f4f6163f034d7376b4b88b94ae49869a3fb83936329ae51b11c8f6cff74 WatchSource:0}: Error finding container fe032f4f6163f034d7376b4b88b94ae49869a3fb83936329ae51b11c8f6cff74: Status 404 returned error can't find the container with id fe032f4f6163f034d7376b4b88b94ae49869a3fb83936329ae51b11c8f6cff74 Nov 03 22:30:55.734331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:55.734312 5272 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:30:56.196996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:56.196962 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" event={"ID":"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d","Type":"ContainerStarted","Data":"fe032f4f6163f034d7376b4b88b94ae49869a3fb83936329ae51b11c8f6cff74"} Nov 03 22:30:56.435464 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:56.435434 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:56.447105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:56.447048 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:57.201926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:57.201893 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" event={"ID":"1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d","Type":"ContainerStarted","Data":"8639acc699da34da5c6d70095d233cdac58ea170da5e99d00cd22a5f22759268"} Nov 03 22:30:57.212661 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:57.212637 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" Nov 03 22:30:57.314068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:57.314024 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" podStartSLOduration=1.7091164779999999 podStartE2EDuration="2.314011456s" totalImagesPullingTime="604.894978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:57.312898497 +0000 UTC m=+437.935206804" watchObservedRunningTime="2026-11-03 22:30:57.314011456 +0000 UTC m=+437.936319762" Nov 03 22:30:57.358914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:57.358887 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:30:58.070081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:30:58.070047 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:31:02.470320 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.470283 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp"] Nov 03 22:31:02.473596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.473580 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.532088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.532057 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pcd\" (UniqueName: \"kubernetes.io/projected/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-kube-api-access-85pcd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.532274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.532161 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.532274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.532192 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.532274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.532259 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.532471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.532284 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.532471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.532440 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.568890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.568859 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 22:31:02.570130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.570109 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp"] Nov 03 22:31:02.633109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.633080 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.633109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.633109 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.633319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.633145 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.633319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.633164 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.633319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.633188 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.633319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.633215 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pcd\" (UniqueName: \"kubernetes.io/projected/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-kube-api-access-85pcd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.634065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.633551 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.634065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.634055 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.634266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.634119 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.640131 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.640105 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.640412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.640395 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.690885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.690855 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pcd\" (UniqueName: \"kubernetes.io/projected/ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0-kube-api-access-85pcd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m59rp\" (UID: \"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:02.785035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:02.785007 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:03.154852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:03.154754 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp"] Nov 03 22:31:03.163216 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:03.163185 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2a7cfe_c63c_4ea0_9b6a_8dc5974392a0.slice/crio-0312765c3af8b9e9ecad2a2b24ac965d081e9e6c3854491e411ecc7ecdac39ea WatchSource:0}: Error finding container 0312765c3af8b9e9ecad2a2b24ac965d081e9e6c3854491e411ecc7ecdac39ea: Status 404 returned error can't find the container with id 0312765c3af8b9e9ecad2a2b24ac965d081e9e6c3854491e411ecc7ecdac39ea Nov 03 22:31:03.229116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:03.229081 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" event={"ID":"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0","Type":"ContainerStarted","Data":"0312765c3af8b9e9ecad2a2b24ac965d081e9e6c3854491e411ecc7ecdac39ea"} Nov 03 22:31:04.234689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:04.234654 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" event={"ID":"ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0","Type":"ContainerStarted","Data":"ac40c922996eb6c2ea35a450229601e2e72cf5cfac3b7d19c9450368a49cb7a6"} Nov 03 22:31:04.345331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:04.345272 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" podStartSLOduration=1.738811204 podStartE2EDuration="2.345248562s" totalImagesPullingTime="606.437358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:04.344882552 +0000 UTC m=+444.967190858" watchObservedRunningTime="2026-11-03 22:31:04.345248562 +0000 UTC m=+444.967556868" Nov 03 22:31:04.785528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:04.785478 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:05.358528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:05.358488 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:31:05.372281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:05.372255 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:31:06.253839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:06.253798 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn" Nov 03 22:31:12.785892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:12.785857 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:12.797728 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:12.797705 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:13.278547 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:13.278473 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" Nov 03 22:31:13.570612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:13.570564 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:31:16.061192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.061158 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6"] Nov 03 22:31:16.064873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.064850 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.113857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.113838 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5vrs\"" Nov 03 22:31:16.151346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151321 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151457 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151357 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151457 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151411 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151474 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151508 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151539 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151561 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svq8c\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-kube-api-access-svq8c\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151624 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.151721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.151652 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252517 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252481 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252533 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252552 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252726 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252847 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252824 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252875 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252905 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.252953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252930 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svq8c\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-kube-api-access-svq8c\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.253062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252962 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.253062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.252962 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.253173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.253150 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.253336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.253306 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.253407 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.253350 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.253520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.253496 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.254902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.254882 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.255430 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.255409 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.265552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.265530 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6"] Nov 03 22:31:16.311296 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.311226 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.311824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.311792 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svq8c\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-kube-api-access-svq8c\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n2sn6\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.377505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.377480 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:16.799911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:16.799883 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6"] Nov 03 22:31:16.807243 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:16.807213 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a26eabd_5f49_4d26_9937_aa599ad02fd8.slice/crio-9f16cdad15c18a8b39a1803414e40664c1bf4fb0ec6551533b8daf4f61a14a67 WatchSource:0}: Error finding container 9f16cdad15c18a8b39a1803414e40664c1bf4fb0ec6551533b8daf4f61a14a67: Status 404 returned error can't find the container with id 9f16cdad15c18a8b39a1803414e40664c1bf4fb0ec6551533b8daf4f61a14a67 Nov 03 22:31:17.285620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:17.285591 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" event={"ID":"8a26eabd-5f49-4d26-9937-aa599ad02fd8","Type":"ContainerStarted","Data":"9f16cdad15c18a8b39a1803414e40664c1bf4fb0ec6551533b8daf4f61a14a67"} Nov 03 22:31:21.429748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:21.429715 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:21.430072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:21.429784 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:21.430072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:21.429828 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:22.308969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:22.308930 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" event={"ID":"8a26eabd-5f49-4d26-9937-aa599ad02fd8","Type":"ContainerStarted","Data":"58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada"} Nov 03 22:31:22.378165 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:22.378126 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:22.379612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:22.379585 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n2sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:31:22.379670 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:22.379653 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 22:31:22.477309 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:22.477243 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podStartSLOduration=1.857455028 podStartE2EDuration="6.477229544s" totalImagesPullingTime="4.619774516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:22.468190888 +0000 UTC m=+463.090499197" watchObservedRunningTime="2026-11-03 22:31:22.477229544 +0000 UTC m=+463.099537853" Nov 03 22:31:23.378659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.378627 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n2sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:31:23.378851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.378683 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 22:31:23.431195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.431162 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77d677f8bf-cvr95"] Nov 03 22:31:23.434555 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.434495 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.491236 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.491205 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77d677f8bf-cvr95"] Nov 03 22:31:23.514452 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.514420 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86d55195-a812-4525-bc69-1419a63288b9-tls-cert\") pod \"authorino-77d677f8bf-cvr95\" (UID: \"86d55195-a812-4525-bc69-1419a63288b9\") " pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.514620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.514472 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjqm\" (UniqueName: \"kubernetes.io/projected/86d55195-a812-4525-bc69-1419a63288b9-kube-api-access-cvjqm\") pod \"authorino-77d677f8bf-cvr95\" (UID: \"86d55195-a812-4525-bc69-1419a63288b9\") " pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.605576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.605542 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-bmsxw"] Nov 03 22:31:23.609030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.609006 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:23.615444 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.615282 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86d55195-a812-4525-bc69-1419a63288b9-tls-cert\") pod \"authorino-77d677f8bf-cvr95\" (UID: \"86d55195-a812-4525-bc69-1419a63288b9\") " pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.615444 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.615362 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjqm\" (UniqueName: \"kubernetes.io/projected/86d55195-a812-4525-bc69-1419a63288b9-kube-api-access-cvjqm\") pod \"authorino-77d677f8bf-cvr95\" (UID: \"86d55195-a812-4525-bc69-1419a63288b9\") " pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.617823 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.617779 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/86d55195-a812-4525-bc69-1419a63288b9-tls-cert\") pod \"authorino-77d677f8bf-cvr95\" (UID: \"86d55195-a812-4525-bc69-1419a63288b9\") " pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.666172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.666081 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-bmsxw"] Nov 03 22:31:23.667867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.666614 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 22:31:23.668013 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.667960 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 22:31:23.668013 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.668002 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-8sx6c\"" Nov 03 22:31:23.673218 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.673196 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjqm\" (UniqueName: \"kubernetes.io/projected/86d55195-a812-4525-bc69-1419a63288b9-kube-api-access-cvjqm\") pod \"authorino-77d677f8bf-cvr95\" (UID: \"86d55195-a812-4525-bc69-1419a63288b9\") " pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.716324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.716284 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5kg4\" (UniqueName: \"kubernetes.io/projected/e1faea8f-ce00-41bf-b473-625d77ca0f04-kube-api-access-p5kg4\") pod \"ai-gateway-controller-74f5787476-bmsxw\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:23.716487 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.716376 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1faea8f-ce00-41bf-b473-625d77ca0f04-metrics-certs\") pod \"ai-gateway-controller-74f5787476-bmsxw\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:23.768109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.768068 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77d677f8bf-cvr95" Nov 03 22:31:23.819176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.818619 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5kg4\" (UniqueName: \"kubernetes.io/projected/e1faea8f-ce00-41bf-b473-625d77ca0f04-kube-api-access-p5kg4\") pod \"ai-gateway-controller-74f5787476-bmsxw\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:23.819176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.818687 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1faea8f-ce00-41bf-b473-625d77ca0f04-metrics-certs\") pod \"ai-gateway-controller-74f5787476-bmsxw\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:23.824445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.824418 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1faea8f-ce00-41bf-b473-625d77ca0f04-metrics-certs\") pod \"ai-gateway-controller-74f5787476-bmsxw\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:23.876745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.876710 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5kg4\" (UniqueName: \"kubernetes.io/projected/e1faea8f-ce00-41bf-b473-625d77ca0f04-kube-api-access-p5kg4\") pod \"ai-gateway-controller-74f5787476-bmsxw\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:23.921664 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:23.921577 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:24.138593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:24.138557 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77d677f8bf-cvr95"] Nov 03 22:31:24.168106 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:24.168062 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86d55195_a812_4525_bc69_1419a63288b9.slice/crio-33682916e7881df26593f67b1b9bb70d0ed466830ed482648600c03fe467cd24 WatchSource:0}: Error finding container 33682916e7881df26593f67b1b9bb70d0ed466830ed482648600c03fe467cd24: Status 404 returned error can't find the container with id 33682916e7881df26593f67b1b9bb70d0ed466830ed482648600c03fe467cd24 Nov 03 22:31:24.312583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:24.312550 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-bmsxw"] Nov 03 22:31:24.320056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:24.320012 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77d677f8bf-cvr95" event={"ID":"86d55195-a812-4525-bc69-1419a63288b9","Type":"ContainerStarted","Data":"33682916e7881df26593f67b1b9bb70d0ed466830ed482648600c03fe467cd24"} Nov 03 22:31:24.320189 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:24.320163 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1faea8f_ce00_41bf_b473_625d77ca0f04.slice/crio-11cb2b423d20b499e1fa9bd7215adaa78991442f87f53290d73e40331ba9ad41 WatchSource:0}: Error finding container 11cb2b423d20b499e1fa9bd7215adaa78991442f87f53290d73e40331ba9ad41: Status 404 returned error can't find the container with id 11cb2b423d20b499e1fa9bd7215adaa78991442f87f53290d73e40331ba9ad41 Nov 03 22:31:24.378789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:24.378750 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n2sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:31:24.378984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:24.378820 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 22:31:24.499144 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:31:24.499108 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:31:25.328813 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:25.328757 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" event={"ID":"e1faea8f-ce00-41bf-b473-625d77ca0f04","Type":"ContainerStarted","Data":"11cb2b423d20b499e1fa9bd7215adaa78991442f87f53290d73e40331ba9ad41"} Nov 03 22:31:25.378476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:25.378105 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n2sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.378476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:25.378165 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 22:31:26.334611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.334546 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77d677f8bf-cvr95" event={"ID":"86d55195-a812-4525-bc69-1419a63288b9","Type":"ContainerStarted","Data":"d9712bc3b54ba0fd357884e89d32f29932c1d75a0377e874b9b3d639781665dc"} Nov 03 22:31:26.378286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.378210 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n2sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.378286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.378265 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 22:31:26.442263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.442207 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77d677f8bf-cvr95" podStartSLOduration=1.489198032 podStartE2EDuration="3.442189597s" totalImagesPullingTime="1.952991565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:26.441257601 +0000 UTC m=+467.063565909" watchObservedRunningTime="2026-11-03 22:31:26.442189597 +0000 UTC m=+467.064497906" Nov 03 22:31:26.477567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.477535 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68cc986d58-f747q"] Nov 03 22:31:26.477937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.477746 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68cc986d58-f747q" podUID="7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" containerName="authorino" containerID="cri-o://ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939" gracePeriod=30 Nov 03 22:31:26.781731 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.781699 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:31:26.872348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.872206 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-kube-api-access-5nzgs\" (UniqueName: \"kubernetes.io/projected/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-kube-api-access-5nzgs\") pod \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " Nov 03 22:31:26.872549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.872404 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-tls-cert\") pod \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\" (UID: \"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692\") " Nov 03 22:31:26.880263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.880228 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-kube-api-access-5nzgs" pod "7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" (UID: "7ac61a7e-338b-4f0e-8950-8a4cf4eb1692"). InnerVolumeSpecName "kube-api-access-5nzgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:31:26.890966 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.890933 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-tls-cert" pod "7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" (UID: "7ac61a7e-338b-4f0e-8950-8a4cf4eb1692"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:31:26.973906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.973733 5272 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-tls-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:31:26.973906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:26.973759 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nzgs\" (UniqueName: \"kubernetes.io/projected/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692-kube-api-access-5nzgs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:31:27.341592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.341552 5272 generic.go:356] "Generic (PLEG): container finished" podID="7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" containerID="ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939" exitCode=0 Nov 03 22:31:27.342181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.341616 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68cc986d58-f747q" Nov 03 22:31:27.342181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.341631 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cc986d58-f747q" event={"ID":"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692","Type":"ContainerDied","Data":"ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939"} Nov 03 22:31:27.342181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.341679 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68cc986d58-f747q" event={"ID":"7ac61a7e-338b-4f0e-8950-8a4cf4eb1692","Type":"ContainerDied","Data":"4b99041c429384e83bcfeb4cf0297979019478838308bbb98423bae33e1d2c00"} Nov 03 22:31:27.342181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.341702 5272 scope.go:122] "RemoveContainer" containerID="ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939" Nov 03 22:31:27.371320 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.371300 5272 scope.go:122] "RemoveContainer" containerID="ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939" Nov 03 22:31:27.372549 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:31:27.372516 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939\": container with ID starting with ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939 not found: ID does not exist" containerID="ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939" Nov 03 22:31:27.372648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.372559 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939"} err="failed to get container status \"ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939\": rpc error: code = NotFound desc = could not find container \"ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939\": container with ID starting with ef960cdba3865c7f50aa7ee8740c0fa3f73e6634955b1c96ba5d632305227939 not found: ID does not exist" Nov 03 22:31:27.378171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.378030 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n2sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:31:27.378171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.378084 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 22:31:27.499403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.499130 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68cc986d58-f747q"] Nov 03 22:31:27.545048 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:27.545017 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68cc986d58-f747q"] Nov 03 22:31:28.378979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:28.378937 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n2sn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.379492 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:28.379004 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 03 22:31:29.494215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:29.494172 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" path="/var/lib/kubelet/pods/7ac61a7e-338b-4f0e-8950-8a4cf4eb1692/volumes" Nov 03 22:31:30.134042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:30.134004 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:30.134972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:30.134793 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:30.141090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:30.141054 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:31:30.363129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:30.363091 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" event={"ID":"e1faea8f-ce00-41bf-b473-625d77ca0f04","Type":"ContainerStarted","Data":"89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4"} Nov 03 22:31:30.585044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:30.584982 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" podStartSLOduration=2.15498722 podStartE2EDuration="7.584962954s" totalImagesPullingTime="5.429975734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:30.583997222 +0000 UTC m=+471.206305544" watchObservedRunningTime="2026-11-03 22:31:30.584962954 +0000 UTC m=+471.207271261" Nov 03 22:31:31.181615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.181580 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl"] Nov 03 22:31:31.182055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.182036 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" containerName="authorino" Nov 03 22:31:31.182164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.182056 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" containerName="authorino" Nov 03 22:31:31.182164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.182152 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac61a7e-338b-4f0e-8950-8a4cf4eb1692" containerName="authorino" Nov 03 22:31:31.185245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.185216 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.277355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.277319 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl"] Nov 03 22:31:31.321891 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.321854 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.321891 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.321894 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed051394-e221-4f2a-bf68-988385bea856-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.322152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.321912 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5c67\" (UniqueName: \"kubernetes.io/projected/ed051394-e221-4f2a-bf68-988385bea856-kube-api-access-q5c67\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.322152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.321929 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.322152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.322046 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.322152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.322079 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.322152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.322118 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed051394-e221-4f2a-bf68-988385bea856-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.322349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.322186 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed051394-e221-4f2a-bf68-988385bea856-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.322349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.322277 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.368636 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.368601 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:31.423168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423131 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423193 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423219 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed051394-e221-4f2a-bf68-988385bea856-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423242 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5c67\" (UniqueName: \"kubernetes.io/projected/ed051394-e221-4f2a-bf68-988385bea856-kube-api-access-q5c67\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423268 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423318 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423342 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423400 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed051394-e221-4f2a-bf68-988385bea856-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423433 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed051394-e221-4f2a-bf68-988385bea856-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423618 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.423840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423681 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.424056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.423952 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.424175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.424154 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.424344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.424324 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed051394-e221-4f2a-bf68-988385bea856-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.425896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.425877 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed051394-e221-4f2a-bf68-988385bea856-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.425991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.425969 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed051394-e221-4f2a-bf68-988385bea856-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.481197 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.481126 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed051394-e221-4f2a-bf68-988385bea856-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.481323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.481203 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5c67\" (UniqueName: \"kubernetes.io/projected/ed051394-e221-4f2a-bf68-988385bea856-kube-api-access-q5c67\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rlkdl\" (UID: \"ed051394-e221-4f2a-bf68-988385bea856\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.498156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.498133 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:31.924936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.924904 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl"] Nov 03 22:31:31.932198 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:31.932169 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded051394_e221_4f2a_bf68_988385bea856.slice/crio-933e72cf7a9fd9c6f73f06afaa4530f59573b57d13132b9ad926c70df67efe89 WatchSource:0}: Error finding container 933e72cf7a9fd9c6f73f06afaa4530f59573b57d13132b9ad926c70df67efe89: Status 404 returned error can't find the container with id 933e72cf7a9fd9c6f73f06afaa4530f59573b57d13132b9ad926c70df67efe89 Nov 03 22:31:31.936005 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.935968 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:31.936082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.936042 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:31.936082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:31.936075 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:32.383317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.381404 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" event={"ID":"ed051394-e221-4f2a-bf68-988385bea856","Type":"ContainerStarted","Data":"dde2e8bf0443666311a680b22aaa587db1a011acafd60c47a1cbaf181df58f67"} Nov 03 22:31:32.383317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.381464 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" event={"ID":"ed051394-e221-4f2a-bf68-988385bea856","Type":"ContainerStarted","Data":"933e72cf7a9fd9c6f73f06afaa4530f59573b57d13132b9ad926c70df67efe89"} Nov 03 22:31:32.494780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.494728 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podStartSLOduration=1.494714277 podStartE2EDuration="1.494714277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:32.493797246 +0000 UTC m=+473.116105557" watchObservedRunningTime="2026-11-03 22:31:32.494714277 +0000 UTC m=+473.117022584" Nov 03 22:31:32.498869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.498847 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:32.499942 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.499923 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:31:32.500003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.499967 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 22:31:32.921559 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.921522 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd"] Nov 03 22:31:32.925108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.925083 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:32.974040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.974004 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 22:31:32.974244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.974085 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tk8gs\"" Nov 03 22:31:32.980460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:32.980427 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd"] Nov 03 22:31:33.045829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.041273 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29pw\" (UniqueName: \"kubernetes.io/projected/784bc8ce-c36d-4bcf-adc5-c58564276bea-kube-api-access-g29pw\") pod \"payload-pre-processing-7b98bb678c-wlsbd\" (UID: \"784bc8ce-c36d-4bcf-adc5-c58564276bea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:33.045829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.041329 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784bc8ce-c36d-4bcf-adc5-c58564276bea-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wlsbd\" (UID: \"784bc8ce-c36d-4bcf-adc5-c58564276bea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:33.070824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.070092 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-6xcbp"] Nov 03 22:31:33.082860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.081183 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:33.131836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.131149 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-6xcbp"] Nov 03 22:31:33.145822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.145096 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29pw\" (UniqueName: \"kubernetes.io/projected/784bc8ce-c36d-4bcf-adc5-c58564276bea-kube-api-access-g29pw\") pod \"payload-pre-processing-7b98bb678c-wlsbd\" (UID: \"784bc8ce-c36d-4bcf-adc5-c58564276bea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:33.145822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.145165 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784bc8ce-c36d-4bcf-adc5-c58564276bea-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wlsbd\" (UID: \"784bc8ce-c36d-4bcf-adc5-c58564276bea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:33.173747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.146795 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784bc8ce-c36d-4bcf-adc5-c58564276bea-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wlsbd\" (UID: \"784bc8ce-c36d-4bcf-adc5-c58564276bea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:33.203516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.203486 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29pw\" (UniqueName: \"kubernetes.io/projected/784bc8ce-c36d-4bcf-adc5-c58564276bea-kube-api-access-g29pw\") pod \"payload-pre-processing-7b98bb678c-wlsbd\" (UID: \"784bc8ce-c36d-4bcf-adc5-c58564276bea\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:33.238569 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.238536 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:33.267829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.249322 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43c630f-c23e-4a54-bf40-4b0e9aaf17a7-plugins-config-volume\") pod \"payload-processing-7bb9489676-6xcbp\" (UID: \"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7\") " pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:33.267829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.249403 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfvl\" (UniqueName: \"kubernetes.io/projected/a43c630f-c23e-4a54-bf40-4b0e9aaf17a7-kube-api-access-lsfvl\") pod \"payload-processing-7bb9489676-6xcbp\" (UID: \"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7\") " pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:33.366829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.364309 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43c630f-c23e-4a54-bf40-4b0e9aaf17a7-plugins-config-volume\") pod \"payload-processing-7bb9489676-6xcbp\" (UID: \"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7\") " pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:33.366829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.364387 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfvl\" (UniqueName: \"kubernetes.io/projected/a43c630f-c23e-4a54-bf40-4b0e9aaf17a7-kube-api-access-lsfvl\") pod \"payload-processing-7bb9489676-6xcbp\" (UID: \"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7\") " pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:33.366829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.365752 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43c630f-c23e-4a54-bf40-4b0e9aaf17a7-plugins-config-volume\") pod \"payload-processing-7bb9489676-6xcbp\" (UID: \"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7\") " pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:33.422872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.422832 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfvl\" (UniqueName: \"kubernetes.io/projected/a43c630f-c23e-4a54-bf40-4b0e9aaf17a7-kube-api-access-lsfvl\") pod \"payload-processing-7bb9489676-6xcbp\" (UID: \"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7\") " pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:33.499585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.499486 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:31:33.499585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.499547 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 22:31:33.647830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.643934 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd"] Nov 03 22:31:33.675677 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:33.675634 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784bc8ce_c36d_4bcf_adc5_c58564276bea.slice/crio-0d831ca0a6418dc57e5604b2419ab5602de6f32f8f26006895ee6aec6b538213 WatchSource:0}: Error finding container 0d831ca0a6418dc57e5604b2419ab5602de6f32f8f26006895ee6aec6b538213: Status 404 returned error can't find the container with id 0d831ca0a6418dc57e5604b2419ab5602de6f32f8f26006895ee6aec6b538213 Nov 03 22:31:33.715379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:33.715144 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:34.114335 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:34.114213 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-6xcbp"] Nov 03 22:31:34.124444 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:34.124399 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43c630f_c23e_4a54_bf40_4b0e9aaf17a7.slice/crio-834e230ca5e3f08f1e7564e71c477a469a6af59582abc70229edad5eed20ef0a WatchSource:0}: Error finding container 834e230ca5e3f08f1e7564e71c477a469a6af59582abc70229edad5eed20ef0a: Status 404 returned error can't find the container with id 834e230ca5e3f08f1e7564e71c477a469a6af59582abc70229edad5eed20ef0a Nov 03 22:31:34.396666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:34.396585 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" event={"ID":"784bc8ce-c36d-4bcf-adc5-c58564276bea","Type":"ContainerStarted","Data":"0d831ca0a6418dc57e5604b2419ab5602de6f32f8f26006895ee6aec6b538213"} Nov 03 22:31:34.398327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:34.398292 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" event={"ID":"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7","Type":"ContainerStarted","Data":"834e230ca5e3f08f1e7564e71c477a469a6af59582abc70229edad5eed20ef0a"} Nov 03 22:31:34.498871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:34.498838 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:31:34.499067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:34.498908 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 22:31:35.499594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:35.499455 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:31:35.499594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:35.499521 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 22:31:36.499330 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:36.498918 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:31:36.499330 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:36.499013 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 22:31:37.499398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:37.499359 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:31:37.499919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:37.499435 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 22:31:38.433295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:38.433024 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" event={"ID":"a43c630f-c23e-4a54-bf40-4b0e9aaf17a7","Type":"ContainerStarted","Data":"4d534bdbd8571a1f80418224bc9e08fa8c55cc047e7a24b369dc8a550b48b026"} Nov 03 22:31:38.433295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:38.433149 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:38.434500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:38.434475 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" event={"ID":"784bc8ce-c36d-4bcf-adc5-c58564276bea","Type":"ContainerStarted","Data":"811617805a128a750be30a9cb4e231b6506d9975967d8f031af137b58ebcd500"} Nov 03 22:31:38.434618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:38.434602 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:38.499273 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:38.499237 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 03 22:31:38.499445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:38.499296 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 03 22:31:38.542870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:38.542823 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" podStartSLOduration=1.8438248590000001 podStartE2EDuration="5.542791345s" totalImagesPullingTime="3.698966486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:38.541690273 +0000 UTC m=+479.163998581" watchObservedRunningTime="2026-11-03 22:31:38.542791345 +0000 UTC m=+479.165099652" Nov 03 22:31:40.499263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:40.499231 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:40.499653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:40.499293 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:41.540986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:41.540942 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:31:41.541389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:41.540995 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:31:42.273511 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.273481 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:42.273839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.273819 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:42.274535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.274517 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" Nov 03 22:31:42.383541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.383508 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:42.387195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.387152 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" podStartSLOduration=6.242777295 podStartE2EDuration="10.387138185s" totalImagesPullingTime="4.14436089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:38.65205271 +0000 UTC m=+479.274361019" watchObservedRunningTime="2026-11-03 22:31:42.387138185 +0000 UTC m=+483.009446492" Nov 03 22:31:42.739540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.739505 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-77fffc8496-hf9sf"] Nov 03 22:31:42.743798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.743778 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:42.769187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.769157 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716b498c-8fc0-4470-a731-689814b925d5-metrics-certs\") pod \"ai-gateway-controller-77fffc8496-hf9sf\" (UID: \"716b498c-8fc0-4470-a731-689814b925d5\") " pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:42.769187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.769190 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdbm\" (UniqueName: \"kubernetes.io/projected/716b498c-8fc0-4470-a731-689814b925d5-kube-api-access-xwdbm\") pod \"ai-gateway-controller-77fffc8496-hf9sf\" (UID: \"716b498c-8fc0-4470-a731-689814b925d5\") " pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:42.797675 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.797645 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-77fffc8496-hf9sf"] Nov 03 22:31:42.869739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.869703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716b498c-8fc0-4470-a731-689814b925d5-metrics-certs\") pod \"ai-gateway-controller-77fffc8496-hf9sf\" (UID: \"716b498c-8fc0-4470-a731-689814b925d5\") " pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:42.869947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.869744 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdbm\" (UniqueName: \"kubernetes.io/projected/716b498c-8fc0-4470-a731-689814b925d5-kube-api-access-xwdbm\") pod \"ai-gateway-controller-77fffc8496-hf9sf\" (UID: \"716b498c-8fc0-4470-a731-689814b925d5\") " pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:42.872133 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.872112 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716b498c-8fc0-4470-a731-689814b925d5-metrics-certs\") pod \"ai-gateway-controller-77fffc8496-hf9sf\" (UID: \"716b498c-8fc0-4470-a731-689814b925d5\") " pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:42.927074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:42.927046 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdbm\" (UniqueName: \"kubernetes.io/projected/716b498c-8fc0-4470-a731-689814b925d5-kube-api-access-xwdbm\") pod \"ai-gateway-controller-77fffc8496-hf9sf\" (UID: \"716b498c-8fc0-4470-a731-689814b925d5\") " pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:43.056693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:43.056636 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:43.425150 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:43.425057 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-77fffc8496-hf9sf"] Nov 03 22:31:43.432203 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:31:43.432168 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716b498c_8fc0_4470_a731_689814b925d5.slice/crio-bf7a37cc0c36b804edbcdd1b0007ad5261f7ff639e4c12a96f58147043cbf68f WatchSource:0}: Error finding container bf7a37cc0c36b804edbcdd1b0007ad5261f7ff639e4c12a96f58147043cbf68f: Status 404 returned error can't find the container with id bf7a37cc0c36b804edbcdd1b0007ad5261f7ff639e4c12a96f58147043cbf68f Nov 03 22:31:43.459111 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:43.459076 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" event={"ID":"716b498c-8fc0-4470-a731-689814b925d5","Type":"ContainerStarted","Data":"bf7a37cc0c36b804edbcdd1b0007ad5261f7ff639e4c12a96f58147043cbf68f"} Nov 03 22:31:44.464867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:44.464832 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" event={"ID":"716b498c-8fc0-4470-a731-689814b925d5","Type":"ContainerStarted","Data":"a5b8bc482d9d5ac24762288dd02f43dd0c0bc2b0cc8cd078d2e514a264d1c800"} Nov 03 22:31:44.465273 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:44.464941 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:44.574530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:44.574478 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" podStartSLOduration=1.645934046 podStartE2EDuration="2.574462874s" totalImagesPullingTime="928.528828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:44.573502575 +0000 UTC m=+485.195810883" watchObservedRunningTime="2026-11-03 22:31:44.574462874 +0000 UTC m=+485.196771184" Nov 03 22:31:49.442040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:49.442002 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-6xcbp" Nov 03 22:31:49.442419 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:49.442074 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wlsbd" Nov 03 22:31:54.499580 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:31:54.499538 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:31:55.471949 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.471918 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-77fffc8496-hf9sf" Nov 03 22:31:55.631829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.631774 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-bmsxw"] Nov 03 22:31:55.632319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.632006 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" podUID="e1faea8f-ce00-41bf-b473-625d77ca0f04" containerName="manager" containerID="cri-o://89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4" gracePeriod=30 Nov 03 22:31:55.879388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.879361 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:55.988204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.988165 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1faea8f-ce00-41bf-b473-625d77ca0f04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1faea8f-ce00-41bf-b473-625d77ca0f04-metrics-certs\") pod \"e1faea8f-ce00-41bf-b473-625d77ca0f04\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " Nov 03 22:31:55.988410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.988268 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1faea8f-ce00-41bf-b473-625d77ca0f04-kube-api-access-p5kg4\" (UniqueName: \"kubernetes.io/projected/e1faea8f-ce00-41bf-b473-625d77ca0f04-kube-api-access-p5kg4\") pod \"e1faea8f-ce00-41bf-b473-625d77ca0f04\" (UID: \"e1faea8f-ce00-41bf-b473-625d77ca0f04\") " Nov 03 22:31:55.990724 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.990687 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1faea8f-ce00-41bf-b473-625d77ca0f04-metrics-certs" pod "e1faea8f-ce00-41bf-b473-625d77ca0f04" (UID: "e1faea8f-ce00-41bf-b473-625d77ca0f04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:31:55.990905 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:55.990875 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1faea8f-ce00-41bf-b473-625d77ca0f04-kube-api-access-p5kg4" pod "e1faea8f-ce00-41bf-b473-625d77ca0f04" (UID: "e1faea8f-ce00-41bf-b473-625d77ca0f04"). InnerVolumeSpecName "kube-api-access-p5kg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:31:56.089193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.089164 5272 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1faea8f-ce00-41bf-b473-625d77ca0f04-metrics-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:31:56.089193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.089191 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5kg4\" (UniqueName: \"kubernetes.io/projected/e1faea8f-ce00-41bf-b473-625d77ca0f04-kube-api-access-p5kg4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:31:56.521902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.521819 5272 generic.go:356] "Generic (PLEG): container finished" podID="e1faea8f-ce00-41bf-b473-625d77ca0f04" containerID="89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4" exitCode=0 Nov 03 22:31:56.521902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.521891 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" Nov 03 22:31:56.522127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.521891 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" event={"ID":"e1faea8f-ce00-41bf-b473-625d77ca0f04","Type":"ContainerDied","Data":"89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4"} Nov 03 22:31:56.522127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.522029 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-74f5787476-bmsxw" event={"ID":"e1faea8f-ce00-41bf-b473-625d77ca0f04","Type":"ContainerDied","Data":"11cb2b423d20b499e1fa9bd7215adaa78991442f87f53290d73e40331ba9ad41"} Nov 03 22:31:56.522127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.522051 5272 scope.go:122] "RemoveContainer" containerID="89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4" Nov 03 22:31:56.530852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.530828 5272 scope.go:122] "RemoveContainer" containerID="89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4" Nov 03 22:31:56.531127 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:31:56.531107 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4\": container with ID starting with 89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4 not found: ID does not exist" containerID="89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4" Nov 03 22:31:56.531175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.531137 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4"} err="failed to get container status \"89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4\": rpc error: code = NotFound desc = could not find container \"89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4\": container with ID starting with 89fa05b216cbe2fcf0cfa1834a2006b1beee48fe692064044733c13f6fb893e4 not found: ID does not exist" Nov 03 22:31:56.674848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.674815 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-bmsxw"] Nov 03 22:31:56.721370 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:56.721336 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-74f5787476-bmsxw"] Nov 03 22:31:57.491645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:31:57.491614 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1faea8f-ce00-41bf-b473-625d77ca0f04" path="/var/lib/kubelet/pods/e1faea8f-ce00-41bf-b473-625d77ca0f04/volumes" Nov 03 22:32:01.421083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.421051 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql"] Nov 03 22:32:01.421495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.421474 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e1faea8f-ce00-41bf-b473-625d77ca0f04" containerName="manager" Nov 03 22:32:01.421495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.421486 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1faea8f-ce00-41bf-b473-625d77ca0f04" containerName="manager" Nov 03 22:32:01.421570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.421561 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1faea8f-ce00-41bf-b473-625d77ca0f04" containerName="manager" Nov 03 22:32:01.426859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.426841 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.523860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.523827 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql"] Nov 03 22:32:01.536557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536527 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.536705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536566 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.536705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536589 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.536705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536702 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.536889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536720 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.536889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536795 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.536889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536854 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.536985 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.536958 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f48rg\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-kube-api-access-f48rg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.537023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.537003 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.637522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637483 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.637522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637520 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.637783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637544 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.637783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637572 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.637783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637611 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f48rg\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-kube-api-access-f48rg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.637783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637647 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.637783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637732 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.638090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637769 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.638090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.637858 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.638203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.638099 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.638203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.638186 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.638302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.638208 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.638302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.638286 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.638852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.638823 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.640189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.640170 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.640270 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.640198 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.695327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.695254 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.695455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.695409 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f48rg\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-kube-api-access-f48rg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8mmql\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:01.738604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:01.738564 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:02.163707 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.163670 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql"] Nov 03 22:32:02.171277 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:32:02.171249 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b097f6_6a43_4fc6_9a0c_e7727aa3b2e9.slice/crio-4b0b3a6509a22bbc97718fb9b3663d514796b348cb2cbef8313a21b667e9ec78 WatchSource:0}: Error finding container 4b0b3a6509a22bbc97718fb9b3663d514796b348cb2cbef8313a21b667e9ec78: Status 404 returned error can't find the container with id 4b0b3a6509a22bbc97718fb9b3663d514796b348cb2cbef8313a21b667e9ec78 Nov 03 22:32:02.173338 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.173304 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:02.173413 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.173373 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:02.173413 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.173402 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:02.550882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.550842 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" event={"ID":"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9","Type":"ContainerStarted","Data":"0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c"} Nov 03 22:32:02.551353 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.550910 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" event={"ID":"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9","Type":"ContainerStarted","Data":"4b0b3a6509a22bbc97718fb9b3663d514796b348cb2cbef8313a21b667e9ec78"} Nov 03 22:32:02.670495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.670423 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podStartSLOduration=1.6704020640000001 podStartE2EDuration="1.670402064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:02.669356019 +0000 UTC m=+503.291664350" watchObservedRunningTime="2026-11-03 22:32:02.670402064 +0000 UTC m=+503.292710375" Nov 03 22:32:02.739456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:02.739422 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:03.740914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:03.740884 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8mmql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:32:03.741304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:03.740936 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 22:32:04.739125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:04.739082 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8mmql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:32:04.739325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:04.739143 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 22:32:05.739997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:05.739956 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8mmql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:32:05.740369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:05.740016 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 22:32:06.739063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:06.739025 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8mmql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:32:06.739247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:06.739084 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 22:32:07.739869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:07.739796 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8mmql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:32:07.740259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:07.739885 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 22:32:08.740003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:08.739957 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8mmql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 22:32:08.740383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:08.740034 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 22:32:10.492006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:10.491970 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:10.492517 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:10.492494 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:10.493392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:10.493369 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:32:24.499702 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:32:24.499663 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:32:43.958789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:43.958749 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz"] Nov 03 22:32:43.965229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:43.965185 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.014545 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.014518 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6f1190-openshift-default-dockercfg-fjmzm\"" Nov 03 22:32:44.017948 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.017920 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.017954 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.017971 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.017992 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.018130 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.018194 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.018317 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwn7v\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-kube-api-access-dwn7v\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018443 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.018363 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.018443 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.018387 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.062290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.062247 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz"] Nov 03 22:32:44.119904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.119862 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.119969 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwn7v\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-kube-api-access-dwn7v\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120003 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120029 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120124 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120154 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120177 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120206 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120236 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120518 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120367 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.120681 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120654 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.121023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.120995 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.122020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.121372 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.122539 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.122509 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.123463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.123438 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.124018 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.123993 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.177833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.177775 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.178585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.178556 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwn7v\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-kube-api-access-dwn7v\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.279830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.279724 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:44.574526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.574486 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt"] Nov 03 22:32:44.579303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.579282 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.624725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.624687 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.624725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.624730 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc95c\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-kube-api-access-cc95c\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.624972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.624835 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.624972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.624864 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.624972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.624907 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.625105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.625066 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.625105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.625098 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.625182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.625142 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.625223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.625190 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.628034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.628005 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-545b5d-openshift-default-dockercfg-t5wb5\"" Nov 03 22:32:44.636039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.636010 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt"] Nov 03 22:32:44.725886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.725846 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.725895 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.725959 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.725991 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc95c\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-kube-api-access-cc95c\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.726040 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.726081 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.726111 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.726136 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.726162 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.726596 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.726719 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.726697 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.727434 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.727363 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.727564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.727500 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.727928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.727898 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.729924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.729880 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.730044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.729929 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.745091 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.745056 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz"] Nov 03 22:32:44.752423 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:32:44.752394 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd875c132_fdfc_4f22_8167_cbbb8c44fffe.slice/crio-abd446ab5cb1d6c00535b75e7080fc8252d72a5135f7033984ae83cda3725ab6 WatchSource:0}: Error finding container abd446ab5cb1d6c00535b75e7080fc8252d72a5135f7033984ae83cda3725ab6: Status 404 returned error can't find the container with id abd446ab5cb1d6c00535b75e7080fc8252d72a5135f7033984ae83cda3725ab6 Nov 03 22:32:44.754522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.754487 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:44.754598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.754560 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:44.754598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.754586 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:44.784645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.784618 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.784829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.784789 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc95c\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-kube-api-access-cc95c\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:44.893034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:44.892925 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:45.333349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.333314 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt"] Nov 03 22:32:45.352167 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:32:45.352125 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a587b1b_f8dd_41f3_9766_cc48b39a60a9.slice/crio-caf7b245cf6c127a6dcce8e105fad7fb2b0cb8f675a3746dea52951c90e9062b WatchSource:0}: Error finding container caf7b245cf6c127a6dcce8e105fad7fb2b0cb8f675a3746dea52951c90e9062b: Status 404 returned error can't find the container with id caf7b245cf6c127a6dcce8e105fad7fb2b0cb8f675a3746dea52951c90e9062b Nov 03 22:32:45.371473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.371403 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:45.371629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.371517 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:45.371629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.371559 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:45.759364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.759194 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" event={"ID":"2a587b1b-f8dd-41f3-9766-cc48b39a60a9","Type":"ContainerStarted","Data":"3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4"} Nov 03 22:32:45.759364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.759274 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" event={"ID":"2a587b1b-f8dd-41f3-9766-cc48b39a60a9","Type":"ContainerStarted","Data":"caf7b245cf6c127a6dcce8e105fad7fb2b0cb8f675a3746dea52951c90e9062b"} Nov 03 22:32:45.774780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.760983 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" event={"ID":"d875c132-fdfc-4f22-8167-cbbb8c44fffe","Type":"ContainerStarted","Data":"89be454ba77027205e2e7c89dc7c055e03a11c5458dbba424d35c71e3ba28238"} Nov 03 22:32:45.774780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.761039 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" event={"ID":"d875c132-fdfc-4f22-8167-cbbb8c44fffe","Type":"ContainerStarted","Data":"abd446ab5cb1d6c00535b75e7080fc8252d72a5135f7033984ae83cda3725ab6"} Nov 03 22:32:45.878765 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.878712 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podStartSLOduration=1.878696237 podStartE2EDuration="1.878696237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:45.878013762 +0000 UTC m=+546.500322069" watchObservedRunningTime="2026-11-03 22:32:45.878696237 +0000 UTC m=+546.501004544" Nov 03 22:32:45.893770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.893744 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:45.991788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:45.991645 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podStartSLOduration=2.9916269460000002 podStartE2EDuration="2.991626946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:45.991348011 +0000 UTC m=+546.613656316" watchObservedRunningTime="2026-11-03 22:32:45.991626946 +0000 UTC m=+546.613935256" Nov 03 22:32:46.282825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:46.280672 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:46.294876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:46.294823 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:46.766035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:46.765949 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:46.767022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:46.767000 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:32:46.898895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:46.898861 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:47.769860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:47.769823 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:47.770863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:47.770843 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:32:52.874443 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:52.874360 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s"] Nov 03 22:32:52.878001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:52.877979 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:52.914562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:52.914529 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bf81353-20bd-47f2-9dd1-cf95f768a565-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:52.914715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:52.914600 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpwp\" (UniqueName: \"kubernetes.io/projected/5bf81353-20bd-47f2-9dd1-cf95f768a565-kube-api-access-qkpwp\") pod \"payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:52.931269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:52.931242 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s"] Nov 03 22:32:52.985827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:52.984901 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5e72ad\"" Nov 03 22:32:53.015502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.015470 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bf81353-20bd-47f2-9dd1-cf95f768a565-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:53.015698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.015546 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpwp\" (UniqueName: \"kubernetes.io/projected/5bf81353-20bd-47f2-9dd1-cf95f768a565-kube-api-access-qkpwp\") pod \"payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:53.016189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.016168 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bf81353-20bd-47f2-9dd1-cf95f768a565-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:53.024736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.024707 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5e72ad-dockercfg-x8566\"" Nov 03 22:32:53.032812 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.032776 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh"] Nov 03 22:32:53.036384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.036368 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.072568 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.072542 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpwp\" (UniqueName: \"kubernetes.io/projected/5bf81353-20bd-47f2-9dd1-cf95f768a565-kube-api-access-qkpwp\") pod \"payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:53.079753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.079487 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh"] Nov 03 22:32:53.116495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.116458 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e06ab06-701a-4213-b11e-3e4be2f716af-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.116672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.116515 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nls\" (UniqueName: \"kubernetes.io/projected/1e06ab06-701a-4213-b11e-3e4be2f716af-kube-api-access-t2nls\") pod \"payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.191768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.191673 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:53.217023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.216996 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e06ab06-701a-4213-b11e-3e4be2f716af-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.217129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.217051 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nls\" (UniqueName: \"kubernetes.io/projected/1e06ab06-701a-4213-b11e-3e4be2f716af-kube-api-access-t2nls\") pod \"payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.217697 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.217680 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e06ab06-701a-4213-b11e-3e4be2f716af-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.274412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.274293 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nls\" (UniqueName: \"kubernetes.io/projected/1e06ab06-701a-4213-b11e-3e4be2f716af-kube-api-access-t2nls\") pod \"payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.349711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.349674 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.562702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.562668 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s"] Nov 03 22:32:53.569913 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:32:53.569887 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf81353_20bd_47f2_9dd1_cf95f768a565.slice/crio-13d7fda8b8839a386440334d7882d36c32d2ed3546633dfe078245850b5a7a23 WatchSource:0}: Error finding container 13d7fda8b8839a386440334d7882d36c32d2ed3546633dfe078245850b5a7a23: Status 404 returned error can't find the container with id 13d7fda8b8839a386440334d7882d36c32d2ed3546633dfe078245850b5a7a23 Nov 03 22:32:53.719706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.719671 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh"] Nov 03 22:32:53.727202 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:32:53.727175 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e06ab06_701a_4213_b11e_3e4be2f716af.slice/crio-cb61053c00c08f930b949443e148c476eb9f3b580f9344e81dc88ba4bedaac88 WatchSource:0}: Error finding container cb61053c00c08f930b949443e148c476eb9f3b580f9344e81dc88ba4bedaac88: Status 404 returned error can't find the container with id cb61053c00c08f930b949443e148c476eb9f3b580f9344e81dc88ba4bedaac88 Nov 03 22:32:53.798215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.798173 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" event={"ID":"5bf81353-20bd-47f2-9dd1-cf95f768a565","Type":"ContainerStarted","Data":"0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382"} Nov 03 22:32:53.798375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.798239 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" event={"ID":"5bf81353-20bd-47f2-9dd1-cf95f768a565","Type":"ContainerStarted","Data":"13d7fda8b8839a386440334d7882d36c32d2ed3546633dfe078245850b5a7a23"} Nov 03 22:32:53.798512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.798396 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:32:53.799878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.799851 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" event={"ID":"1e06ab06-701a-4213-b11e-3e4be2f716af","Type":"ContainerStarted","Data":"6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022"} Nov 03 22:32:53.799975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.799899 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:32:53.799975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.799916 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" event={"ID":"1e06ab06-701a-4213-b11e-3e4be2f716af","Type":"ContainerStarted","Data":"cb61053c00c08f930b949443e148c476eb9f3b580f9344e81dc88ba4bedaac88"} Nov 03 22:32:53.907960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:53.907853 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" podStartSLOduration=1.90783733 podStartE2EDuration="1.90783733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:53.907143323 +0000 UTC m=+554.529451628" watchObservedRunningTime="2026-11-03 22:32:53.90783733 +0000 UTC m=+554.530146025" Nov 03 22:32:54.015686 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:54.015626 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" podStartSLOduration=2.015609818 podStartE2EDuration="2.015609818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:54.014602061 +0000 UTC m=+554.636910369" watchObservedRunningTime="2026-11-03 22:32:54.015609818 +0000 UTC m=+554.637918126" Nov 03 22:32:54.500187 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:32:54.500155 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:32:57.931170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:57.931135 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7"] Nov 03 22:32:57.934726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:57.934704 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.029996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.029957 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6f1190\"" Nov 03 22:32:58.030168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.030016 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6f1190-dockercfg-mlqw4\"" Nov 03 22:32:58.031609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.031589 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7"] Nov 03 22:32:58.068024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.067989 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0e8e22-ee36-4264-89f3-36c93ac79c62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.068198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.068075 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbwl\" (UniqueName: \"kubernetes.io/projected/db0e8e22-ee36-4264-89f3-36c93ac79c62-kube-api-access-mkbwl\") pod \"payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.169134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.169096 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0e8e22-ee36-4264-89f3-36c93ac79c62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.169332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.169160 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbwl\" (UniqueName: \"kubernetes.io/projected/db0e8e22-ee36-4264-89f3-36c93ac79c62-kube-api-access-mkbwl\") pod \"payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.170224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.170192 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0e8e22-ee36-4264-89f3-36c93ac79c62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.227510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.227428 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbwl\" (UniqueName: \"kubernetes.io/projected/db0e8e22-ee36-4264-89f3-36c93ac79c62-kube-api-access-mkbwl\") pod \"payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.247692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.247657 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.620324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.620278 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7"] Nov 03 22:32:58.627855 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:32:58.627781 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0e8e22_ee36_4264_89f3_36c93ac79c62.slice/crio-f562e3710b962ff06e8425a8edfb2cd072090cde8eb723610656145c6ec9f7f1 WatchSource:0}: Error finding container f562e3710b962ff06e8425a8edfb2cd072090cde8eb723610656145c6ec9f7f1: Status 404 returned error can't find the container with id f562e3710b962ff06e8425a8edfb2cd072090cde8eb723610656145c6ec9f7f1 Nov 03 22:32:58.824028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.823987 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" event={"ID":"db0e8e22-ee36-4264-89f3-36c93ac79c62","Type":"ContainerStarted","Data":"6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2"} Nov 03 22:32:58.824217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.824055 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" event={"ID":"db0e8e22-ee36-4264-89f3-36c93ac79c62","Type":"ContainerStarted","Data":"f562e3710b962ff06e8425a8edfb2cd072090cde8eb723610656145c6ec9f7f1"} Nov 03 22:32:58.824217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.824085 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:32:58.933616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:32:58.933513 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" podStartSLOduration=1.9334975 podStartE2EDuration="1.9334975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:58.93136868 +0000 UTC m=+559.553676987" watchObservedRunningTime="2026-11-03 22:32:58.9334975 +0000 UTC m=+559.555805807" Nov 03 22:33:00.288136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.288096 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5"] Nov 03 22:33:00.292161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.292138 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.389447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.389417 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ee74cb-dockercfg-jnjzj\"" Nov 03 22:33:00.389447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.389417 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ee74cb\"" Nov 03 22:33:00.391205 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.391184 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5"] Nov 03 22:33:00.391578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.391559 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299f09cd-663e-4790-8f75-8f1f2698571c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.391616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.391606 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzqj\" (UniqueName: \"kubernetes.io/projected/299f09cd-663e-4790-8f75-8f1f2698571c-kube-api-access-9mzqj\") pod \"payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.439361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.439327 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss"] Nov 03 22:33:00.443088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.443065 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.495397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.495359 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299f09cd-663e-4790-8f75-8f1f2698571c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.495698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.495675 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzqj\" (UniqueName: \"kubernetes.io/projected/299f09cd-663e-4790-8f75-8f1f2698571c-kube-api-access-9mzqj\") pod \"payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.513057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.513004 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8hv\" (UniqueName: \"kubernetes.io/projected/fe803d4b-d693-4f97-9f1c-c495aaea0276-kube-api-access-ck8hv\") pod \"payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.513428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.513401 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe803d4b-d693-4f97-9f1c-c495aaea0276-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.513732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.496655 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299f09cd-663e-4790-8f75-8f1f2698571c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.516121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.515765 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss"] Nov 03 22:33:00.562630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.562566 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzqj\" (UniqueName: \"kubernetes.io/projected/299f09cd-663e-4790-8f75-8f1f2698571c-kube-api-access-9mzqj\") pod \"payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.605212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.605176 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:00.614714 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.614685 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8hv\" (UniqueName: \"kubernetes.io/projected/fe803d4b-d693-4f97-9f1c-c495aaea0276-kube-api-access-ck8hv\") pod \"payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.614846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.614727 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe803d4b-d693-4f97-9f1c-c495aaea0276-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.615362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.615334 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe803d4b-d693-4f97-9f1c-c495aaea0276-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.673033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.673007 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8hv\" (UniqueName: \"kubernetes.io/projected/fe803d4b-d693-4f97-9f1c-c495aaea0276-kube-api-access-ck8hv\") pod \"payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.757017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.756980 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:00.976842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:00.976743 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5"] Nov 03 22:33:00.983562 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:00.983534 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299f09cd_663e_4790_8f75_8f1f2698571c.slice/crio-d967d27aaebfbbc919d5f982c96b47d75cddc244344c9944f3399e2ce2e0e115 WatchSource:0}: Error finding container d967d27aaebfbbc919d5f982c96b47d75cddc244344c9944f3399e2ce2e0e115: Status 404 returned error can't find the container with id d967d27aaebfbbc919d5f982c96b47d75cddc244344c9944f3399e2ce2e0e115 Nov 03 22:33:01.125027 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.124991 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss"] Nov 03 22:33:01.132068 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:01.132040 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe803d4b_d693_4f97_9f1c_c495aaea0276.slice/crio-90d190aee65b3bf3f102dad9daa0e67f65802aa17532638a643cce54793920f3 WatchSource:0}: Error finding container 90d190aee65b3bf3f102dad9daa0e67f65802aa17532638a643cce54793920f3: Status 404 returned error can't find the container with id 90d190aee65b3bf3f102dad9daa0e67f65802aa17532638a643cce54793920f3 Nov 03 22:33:01.842510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.842468 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" event={"ID":"fe803d4b-d693-4f97-9f1c-c495aaea0276","Type":"ContainerStarted","Data":"dfaa09cfebcb16909b13d36558bce92d205ac645d57ae3f608d079e861d24b88"} Nov 03 22:33:01.842956 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.842620 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:01.842956 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.842670 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" event={"ID":"fe803d4b-d693-4f97-9f1c-c495aaea0276","Type":"ContainerStarted","Data":"90d190aee65b3bf3f102dad9daa0e67f65802aa17532638a643cce54793920f3"} Nov 03 22:33:01.844056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.844028 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" event={"ID":"299f09cd-663e-4790-8f75-8f1f2698571c","Type":"ContainerStarted","Data":"559668611da7931d5e7aa5ccd1085cae651bd1030a0bee233be184a56b399d21"} Nov 03 22:33:01.844203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.844077 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" event={"ID":"299f09cd-663e-4790-8f75-8f1f2698571c","Type":"ContainerStarted","Data":"d967d27aaebfbbc919d5f982c96b47d75cddc244344c9944f3399e2ce2e0e115"} Nov 03 22:33:01.844203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.844165 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:01.952074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:01.952027 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" podStartSLOduration=1.9520121879999999 podStartE2EDuration="1.952012188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:01.950063301 +0000 UTC m=+562.572371642" watchObservedRunningTime="2026-11-03 22:33:01.952012188 +0000 UTC m=+562.574320495" Nov 03 22:33:02.058435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.058271 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" podStartSLOduration=2.058253965 podStartE2EDuration="2.058253965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:02.057695268 +0000 UTC m=+562.680003576" watchObservedRunningTime="2026-11-03 22:33:02.058253965 +0000 UTC m=+562.680562277" Nov 03 22:33:02.851205 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.851169 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw"] Nov 03 22:33:02.854527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.854506 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:02.929393 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.929364 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-68fbd1\"" Nov 03 22:33:02.933410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.933299 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw"] Nov 03 22:33:02.952667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.952626 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad38c13-2076-468f-ac66-85718c705402-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:02.952931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.952724 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzbv\" (UniqueName: \"kubernetes.io/projected/2ad38c13-2076-468f-ac66-85718c705402-kube-api-access-nxzbv\") pod \"payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:02.972147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:02.972111 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-68fbd1-dockercfg-gz7cf\"" Nov 03 22:33:03.004743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.004711 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z"] Nov 03 22:33:03.008836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.008813 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.053974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.053939 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzbv\" (UniqueName: \"kubernetes.io/projected/2ad38c13-2076-468f-ac66-85718c705402-kube-api-access-nxzbv\") pod \"payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:03.054158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.054048 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad38c13-2076-468f-ac66-85718c705402-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:03.054679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.054658 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad38c13-2076-468f-ac66-85718c705402-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:03.061937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.061905 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z"] Nov 03 22:33:03.111900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.111798 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzbv\" (UniqueName: \"kubernetes.io/projected/2ad38c13-2076-468f-ac66-85718c705402-kube-api-access-nxzbv\") pod \"payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:03.155245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.155208 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kztbx\" (UniqueName: \"kubernetes.io/projected/57cb4792-4340-4f77-995a-8eebd5c02792-kube-api-access-kztbx\") pod \"payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.155419 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.155292 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb4792-4340-4f77-995a-8eebd5c02792-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.167594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.167556 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:03.265871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.265833 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kztbx\" (UniqueName: \"kubernetes.io/projected/57cb4792-4340-4f77-995a-8eebd5c02792-kube-api-access-kztbx\") pod \"payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.266072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.265915 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb4792-4340-4f77-995a-8eebd5c02792-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.275859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.271253 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb4792-4340-4f77-995a-8eebd5c02792-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.326753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.326716 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kztbx\" (UniqueName: \"kubernetes.io/projected/57cb4792-4340-4f77-995a-8eebd5c02792-kube-api-access-kztbx\") pod \"payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.544205 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.544172 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw"] Nov 03 22:33:03.550840 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:03.550797 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad38c13_2076_468f_ac66_85718c705402.slice/crio-c62e30e8d79cab70a3b0f5189cacc835e4e9447b640ad9db866d3a18a89751b8 WatchSource:0}: Error finding container c62e30e8d79cab70a3b0f5189cacc835e4e9447b640ad9db866d3a18a89751b8: Status 404 returned error can't find the container with id c62e30e8d79cab70a3b0f5189cacc835e4e9447b640ad9db866d3a18a89751b8 Nov 03 22:33:03.621694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.621669 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:03.855915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.855797 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" event={"ID":"2ad38c13-2076-468f-ac66-85718c705402","Type":"ContainerStarted","Data":"ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d"} Nov 03 22:33:03.856287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.855916 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" event={"ID":"2ad38c13-2076-468f-ac66-85718c705402","Type":"ContainerStarted","Data":"c62e30e8d79cab70a3b0f5189cacc835e4e9447b640ad9db866d3a18a89751b8"} Nov 03 22:33:03.856287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.855945 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:03.965425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.965364 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" podStartSLOduration=1.96534732 podStartE2EDuration="1.96534732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:03.964866577 +0000 UTC m=+564.587174888" watchObservedRunningTime="2026-11-03 22:33:03.96534732 +0000 UTC m=+564.587655629" Nov 03 22:33:03.989784 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:03.989746 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z"] Nov 03 22:33:03.997451 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:03.997412 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cb4792_4340_4f77_995a_8eebd5c02792.slice/crio-43f0a90c112c2792053446cf24a2542bd917ceb4e3ddd5a27dda53b6eefc9171 WatchSource:0}: Error finding container 43f0a90c112c2792053446cf24a2542bd917ceb4e3ddd5a27dda53b6eefc9171: Status 404 returned error can't find the container with id 43f0a90c112c2792053446cf24a2542bd917ceb4e3ddd5a27dda53b6eefc9171 Nov 03 22:33:04.807688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:04.807650 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 22:33:04.807928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:04.807729 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 22:33:04.870964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:04.870927 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" event={"ID":"57cb4792-4340-4f77-995a-8eebd5c02792","Type":"ContainerStarted","Data":"ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4"} Nov 03 22:33:04.871588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:04.871503 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" event={"ID":"57cb4792-4340-4f77-995a-8eebd5c02792","Type":"ContainerStarted","Data":"43f0a90c112c2792053446cf24a2542bd917ceb4e3ddd5a27dda53b6eefc9171"} Nov 03 22:33:04.871771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:04.871746 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:05.082349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:05.082256 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:33:05.143361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:05.143294 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" podStartSLOduration=3.143272928 podStartE2EDuration="3.143272928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:05.142270515 +0000 UTC m=+565.764578820" watchObservedRunningTime="2026-11-03 22:33:05.143272928 +0000 UTC m=+565.765581238" Nov 03 22:33:08.005544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.004978 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh"] Nov 03 22:33:08.010853 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.010828 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.087203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.087156 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh"] Nov 03 22:33:08.119907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.119877 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-961054\"" Nov 03 22:33:08.119907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.119885 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-961054-dockercfg-6kmx7\"" Nov 03 22:33:08.125481 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.125451 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52wn\" (UniqueName: \"kubernetes.io/projected/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-kube-api-access-k52wn\") pod \"payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.125616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.125596 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.226600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.226563 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52wn\" (UniqueName: \"kubernetes.io/projected/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-kube-api-access-k52wn\") pod \"payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.226851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.226662 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.227370 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.227348 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.284379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.284353 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52wn\" (UniqueName: \"kubernetes.io/projected/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-kube-api-access-k52wn\") pod \"payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.325182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.325152 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:08.720581 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.720493 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh"] Nov 03 22:33:08.726925 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:08.726896 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744ce8f2_fa9c_4eb7_a877_3fc09a15e1cd.slice/crio-22215cd852e31f874f8ec58b5338a8c7f9a097887e0cb6e254240bf7275cc71c WatchSource:0}: Error finding container 22215cd852e31f874f8ec58b5338a8c7f9a097887e0cb6e254240bf7275cc71c: Status 404 returned error can't find the container with id 22215cd852e31f874f8ec58b5338a8c7f9a097887e0cb6e254240bf7275cc71c Nov 03 22:33:08.890222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.890187 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" event={"ID":"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd","Type":"ContainerStarted","Data":"da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d"} Nov 03 22:33:08.890410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.890240 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" event={"ID":"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd","Type":"ContainerStarted","Data":"22215cd852e31f874f8ec58b5338a8c7f9a097887e0cb6e254240bf7275cc71c"} Nov 03 22:33:08.890410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:08.890366 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:09.002066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:09.001959 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" podStartSLOduration=2.001945952 podStartE2EDuration="2.001945952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:09.000953867 +0000 UTC m=+569.623262174" watchObservedRunningTime="2026-11-03 22:33:09.001945952 +0000 UTC m=+569.624254259" Nov 03 22:33:09.834008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:09.833973 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 22:33:12.852470 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:12.852440 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 22:33:12.852978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:12.852509 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 22:33:14.170198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.170161 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:33:14.873911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.873832 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 22:33:14.941345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.941305 5272 generic.go:356] "Generic (PLEG): container finished" podID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerID="89be454ba77027205e2e7c89dc7c055e03a11c5458dbba424d35c71e3ba28238" exitCode=0 Nov 03 22:33:14.941727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.941435 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" event={"ID":"d875c132-fdfc-4f22-8167-cbbb8c44fffe","Type":"ContainerDied","Data":"89be454ba77027205e2e7c89dc7c055e03a11c5458dbba424d35c71e3ba28238"} Nov 03 22:33:14.942001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.941981 5272 scope.go:122] "RemoveContainer" containerID="89be454ba77027205e2e7c89dc7c055e03a11c5458dbba424d35c71e3ba28238" Nov 03 22:33:14.943639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.943608 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:14.943750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.943683 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:14.943883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:14.943856 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:15.351830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.351177 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v"] Nov 03 22:33:15.359012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.358982 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.408889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.408858 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-18c03a\"" Nov 03 22:33:15.409119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.408894 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-18c03a-dockercfg-5pzzz\"" Nov 03 22:33:15.411408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.411377 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v"] Nov 03 22:33:15.441792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.441749 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqxd\" (UniqueName: \"kubernetes.io/projected/426df26b-3a98-4e13-a910-9985cd979aca-kube-api-access-mvqxd\") pod \"payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.442009 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.441908 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426df26b-3a98-4e13-a910-9985cd979aca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.546436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.546391 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqxd\" (UniqueName: \"kubernetes.io/projected/426df26b-3a98-4e13-a910-9985cd979aca-kube-api-access-mvqxd\") pod \"payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.547139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.547113 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426df26b-3a98-4e13-a910-9985cd979aca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.553578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.548199 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426df26b-3a98-4e13-a910-9985cd979aca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.604484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.604414 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqxd\" (UniqueName: \"kubernetes.io/projected/426df26b-3a98-4e13-a910-9985cd979aca-kube-api-access-mvqxd\") pod \"payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.679738 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.679701 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:15.876993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.876908 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 22:33:15.961453 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:15.952774 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" event={"ID":"d875c132-fdfc-4f22-8167-cbbb8c44fffe","Type":"ContainerStarted","Data":"cff628ebda889502123c28fe1b59495d042053682283895b6cdc2572987ee4a0"} Nov 03 22:33:16.054023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.053983 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v"] Nov 03 22:33:16.075604 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:16.075572 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426df26b_3a98_4e13_a910_9985cd979aca.slice/crio-1697beb6b903667b440ff134b5663a2efb9375ad88e82b43557e5c69aeef47bd WatchSource:0}: Error finding container 1697beb6b903667b440ff134b5663a2efb9375ad88e82b43557e5c69aeef47bd: Status 404 returned error can't find the container with id 1697beb6b903667b440ff134b5663a2efb9375ad88e82b43557e5c69aeef47bd Nov 03 22:33:16.280080 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.280044 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:33:16.281307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.281282 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:33:16.281426 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.281330 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:33:16.766978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.766932 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:33:16.970848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.970775 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" event={"ID":"426df26b-3a98-4e13-a910-9985cd979aca","Type":"ContainerStarted","Data":"9ca4f5253a2fad2eff4369ddfdccc6b85e7512cddba46965d8a3954b14bb6242"} Nov 03 22:33:16.971017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.970884 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" event={"ID":"426df26b-3a98-4e13-a910-9985cd979aca","Type":"ContainerStarted","Data":"1697beb6b903667b440ff134b5663a2efb9375ad88e82b43557e5c69aeef47bd"} Nov 03 22:33:16.971137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:16.971117 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:17.089701 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.089256 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" podStartSLOduration=2.08923707 podStartE2EDuration="2.08923707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:17.087859549 +0000 UTC m=+577.710167857" watchObservedRunningTime="2026-11-03 22:33:17.08923707 +0000 UTC m=+577.711545379" Nov 03 22:33:17.280476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.280440 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:33:17.280651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.280512 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:33:17.810971 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.810930 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc"] Nov 03 22:33:17.813965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.813942 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:17.864827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.863596 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-545b5d\"" Nov 03 22:33:17.864827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.863831 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-545b5d-dockercfg-trcmt\"" Nov 03 22:33:17.906868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.900221 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmr7h\" (UniqueName: \"kubernetes.io/projected/e3dd6fdc-d6ea-4c02-83ec-53200433861f-kube-api-access-hmr7h\") pod \"payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:17.906868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.900371 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3dd6fdc-d6ea-4c02-83ec-53200433861f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:17.924833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.923958 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc"] Nov 03 22:33:17.982390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.982334 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb"] Nov 03 22:33:17.985848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:17.985826 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.001516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.001492 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmr7h\" (UniqueName: \"kubernetes.io/projected/e3dd6fdc-d6ea-4c02-83ec-53200433861f-kube-api-access-hmr7h\") pod \"payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:18.001628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.001544 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3dd6fdc-d6ea-4c02-83ec-53200433861f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:18.002162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.002145 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3dd6fdc-d6ea-4c02-83ec-53200433861f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:18.015923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.015897 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb"] Nov 03 22:33:18.070277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.070138 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmr7h\" (UniqueName: \"kubernetes.io/projected/e3dd6fdc-d6ea-4c02-83ec-53200433861f-kube-api-access-hmr7h\") pod \"payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:18.103211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.103120 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltxw\" (UniqueName: \"kubernetes.io/projected/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-kube-api-access-8ltxw\") pod \"payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.103387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.103238 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.125550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.125523 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:18.207415 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.207386 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltxw\" (UniqueName: \"kubernetes.io/projected/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-kube-api-access-8ltxw\") pod \"payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.207541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.207451 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.208467 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.208443 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.271166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.271102 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltxw\" (UniqueName: \"kubernetes.io/projected/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-kube-api-access-8ltxw\") pod \"payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.280628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.280553 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:33:18.280628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.280597 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:33:18.297671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.297647 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:18.525859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.525824 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc"] Nov 03 22:33:18.532733 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:18.532704 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dd6fdc_d6ea_4c02_83ec_53200433861f.slice/crio-e8151dcd22d4484bc698c5bacf398d069eedd5de443a4eefa383092835b3efee WatchSource:0}: Error finding container e8151dcd22d4484bc698c5bacf398d069eedd5de443a4eefa383092835b3efee: Status 404 returned error can't find the container with id e8151dcd22d4484bc698c5bacf398d069eedd5de443a4eefa383092835b3efee Nov 03 22:33:18.703937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.703856 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb"] Nov 03 22:33:18.710886 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:33:18.710859 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b59e60a_6dfb_45ce_acb1_4a4471c8a2c9.slice/crio-967c138165fb8667cfd9a4140d70a205127239694123c0b84f3cbfec63dbe939 WatchSource:0}: Error finding container 967c138165fb8667cfd9a4140d70a205127239694123c0b84f3cbfec63dbe939: Status 404 returned error can't find the container with id 967c138165fb8667cfd9a4140d70a205127239694123c0b84f3cbfec63dbe939 Nov 03 22:33:18.995310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.995211 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" event={"ID":"e3dd6fdc-d6ea-4c02-83ec-53200433861f","Type":"ContainerStarted","Data":"b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819"} Nov 03 22:33:18.995745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.995311 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:18.995745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.995339 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" event={"ID":"e3dd6fdc-d6ea-4c02-83ec-53200433861f","Type":"ContainerStarted","Data":"e8151dcd22d4484bc698c5bacf398d069eedd5de443a4eefa383092835b3efee"} Nov 03 22:33:18.996847 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.996820 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" event={"ID":"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9","Type":"ContainerStarted","Data":"adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b"} Nov 03 22:33:18.996956 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.996867 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" event={"ID":"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9","Type":"ContainerStarted","Data":"967c138165fb8667cfd9a4140d70a205127239694123c0b84f3cbfec63dbe939"} Nov 03 22:33:18.996956 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:18.996889 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:19.124717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:19.124665 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" podStartSLOduration=2.124649499 podStartE2EDuration="2.124649499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:19.123214616 +0000 UTC m=+579.745522936" watchObservedRunningTime="2026-11-03 22:33:19.124649499 +0000 UTC m=+579.746957817" Nov 03 22:33:19.234060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:19.233998 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" podStartSLOduration=2.233983363 podStartE2EDuration="2.233983363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:19.233089634 +0000 UTC m=+579.855397944" watchObservedRunningTime="2026-11-03 22:33:19.233983363 +0000 UTC m=+579.856291670" Nov 03 22:33:19.280926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:19.280830 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:33:19.280926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:19.280890 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:33:19.899078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:19.899047 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 22:33:20.281222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:20.281131 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:33:20.281222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:20.281186 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:33:21.284358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:21.284320 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:33:21.285275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:21.285259 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 22:33:23.182148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:23.182110 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:33:24.500514 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:33:24.500469 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:33:27.986870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:27.986832 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 22:33:30.004868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:30.004842 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 22:33:30.005340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:30.004908 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 22:33:43.501871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:43.501832 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:33:52.603826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:52.601848 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:33:53.205627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:53.205595 5272 generic.go:356] "Generic (PLEG): container finished" podID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerID="3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4" exitCode=0 Nov 03 22:33:53.205866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:53.205669 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" event={"ID":"2a587b1b-f8dd-41f3-9766-cc48b39a60a9","Type":"ContainerDied","Data":"3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4"} Nov 03 22:33:53.206155 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:53.206140 5272 scope.go:122] "RemoveContainer" containerID="3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4" Nov 03 22:33:53.207519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:53.207489 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:53.207586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:53.207560 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:53.207621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:53.207604 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:54.212361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:54.212322 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" event={"ID":"2a587b1b-f8dd-41f3-9766-cc48b39a60a9","Type":"ContainerStarted","Data":"1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba"} Nov 03 22:33:54.501359 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:33:54.501276 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:33:54.893997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:54.893962 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:33:54.894201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:54.894141 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:33:54.894277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:54.894204 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:33:55.893994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:55.893957 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:33:55.894503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:55.894018 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:33:56.894578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:56.894540 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:33:56.895378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:56.895300 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:33:57.893846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:57.893781 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:33:57.894040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:57.893866 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:33:58.894168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:58.894131 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:33:58.894595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:58.894184 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:33:59.898885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:59.898852 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:33:59.899388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:59.899236 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:33:59.900093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:33:59.900072 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 22:34:13.178484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.178444 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:34:13.391579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.391545 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl"] Nov 03 22:34:13.396357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.396327 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.450348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.450249 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-961054\"/\"openshift-service-ca.crt\"" Nov 03 22:34:13.496934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.496900 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-961054\"/\"kube-root-ca.crt\"" Nov 03 22:34:13.497122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.496991 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-961054\"/\"simulator-w3-961054-kserve-self-signed-certs\"" Nov 03 22:34:13.497122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.497088 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-961054\"/\"default-dockercfg-9ssgv\"" Nov 03 22:34:13.498678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.498642 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl"] Nov 03 22:34:13.565422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.564956 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tls-certs\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.565422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.565044 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkb2\" (UniqueName: \"kubernetes.io/projected/1a2d866d-b309-4640-98f7-a8e9e8f94a25-kube-api-access-cnkb2\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.565422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.565095 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-home\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.565422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.565116 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-model-cache\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.565422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.565146 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tmp-dir\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.565422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.565178 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-dshm\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.666928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.666796 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tls-certs\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.666928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.666916 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkb2\" (UniqueName: \"kubernetes.io/projected/1a2d866d-b309-4640-98f7-a8e9e8f94a25-kube-api-access-cnkb2\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.667211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.666968 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-home\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.667211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.666996 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-model-cache\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.667211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.667021 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tmp-dir\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.667373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.667254 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-dshm\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.667550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.667455 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-model-cache\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.667737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.667706 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tmp-dir\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.667900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.667877 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-home\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.669906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.669885 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-dshm\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.670308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.670288 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tls-certs\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:13.724625 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:13.724543 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkb2\" (UniqueName: \"kubernetes.io/projected/1a2d866d-b309-4640-98f7-a8e9e8f94a25-kube-api-access-cnkb2\") pod \"simulator-w3-961054-kserve-84586b6dd7-wkssl\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:14.008580 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:14.008494 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:14.382261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:14.382221 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl"] Nov 03 22:34:14.389578 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:34:14.389538 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a2d866d_b309_4640_98f7_a8e9e8f94a25.slice/crio-5f38a23f8c92e8e9e4aaaffdc02731f4ce1f0bb7892de80d10fa9f74d5b9b7b0 WatchSource:0}: Error finding container 5f38a23f8c92e8e9e4aaaffdc02731f4ce1f0bb7892de80d10fa9f74d5b9b7b0: Status 404 returned error can't find the container with id 5f38a23f8c92e8e9e4aaaffdc02731f4ce1f0bb7892de80d10fa9f74d5b9b7b0 Nov 03 22:34:15.335672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:15.335626 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" event={"ID":"1a2d866d-b309-4640-98f7-a8e9e8f94a25","Type":"ContainerStarted","Data":"e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a"} Nov 03 22:34:15.335672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:15.335678 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" event={"ID":"1a2d866d-b309-4640-98f7-a8e9e8f94a25","Type":"ContainerStarted","Data":"5f38a23f8c92e8e9e4aaaffdc02731f4ce1f0bb7892de80d10fa9f74d5b9b7b0"} Nov 03 22:34:15.446221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:15.446168 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podStartSLOduration=2.446154435 podStartE2EDuration="2.446154435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:15.444610227 +0000 UTC m=+636.066918535" watchObservedRunningTime="2026-11-03 22:34:15.446154435 +0000 UTC m=+636.068462742" Nov 03 22:34:16.008890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:16.008846 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:22.096499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.096416 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw"] Nov 03 22:34:22.100392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.100375 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.159900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.159871 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"default-dockercfg-pkpfl\"" Nov 03 22:34:22.162526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.162498 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-model-cache\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.162663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.162562 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-dshm\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.162729 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.162701 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-home\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.162771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.162748 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tmp-dir\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.162880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.162863 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tls-certs\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.162928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.162908 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jmk\" (UniqueName: \"kubernetes.io/projected/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-kube-api-access-f7jmk\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.195696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.195664 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"simulator-w6-5e72ad-kserve-self-signed-certs\"" Nov 03 22:34:22.195887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.195760 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"openshift-service-ca.crt\"" Nov 03 22:34:22.195887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.195826 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"kube-root-ca.crt\"" Nov 03 22:34:22.197887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.197862 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw"] Nov 03 22:34:22.264087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264055 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jmk\" (UniqueName: \"kubernetes.io/projected/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-kube-api-access-f7jmk\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264101 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-model-cache\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264162 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-dshm\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264229 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-home\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264409 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264260 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tmp-dir\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264409 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264342 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tls-certs\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264569 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-model-cache\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264613 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-home\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.264734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.264697 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tmp-dir\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.266358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.266340 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-dshm\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.266712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.266694 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tls-certs\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.323832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.323773 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jmk\" (UniqueName: \"kubernetes.io/projected/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-kube-api-access-f7jmk\") pod \"simulator-w6-5e72ad-kserve-98c947c5f-7gvvw\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.412607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.412519 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:22.788137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:22.788099 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw"] Nov 03 22:34:22.794403 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:34:22.794375 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0daab0f_a22b_4a3a_b54e_23a6ccf9b1de.slice/crio-7e5d524aed5f0e7bf67ba8a882f223d036f5706593274af2593a0778e99fdc4e WatchSource:0}: Error finding container 7e5d524aed5f0e7bf67ba8a882f223d036f5706593274af2593a0778e99fdc4e: Status 404 returned error can't find the container with id 7e5d524aed5f0e7bf67ba8a882f223d036f5706593274af2593a0778e99fdc4e Nov 03 22:34:23.389247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:23.389213 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" event={"ID":"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de","Type":"ContainerStarted","Data":"bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9"} Nov 03 22:34:23.389653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:23.389258 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" event={"ID":"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de","Type":"ContainerStarted","Data":"7e5d524aed5f0e7bf67ba8a882f223d036f5706593274af2593a0778e99fdc4e"} Nov 03 22:34:23.413009 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:23.412980 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:23.500822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:23.500749 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podStartSLOduration=1.500734885 podStartE2EDuration="1.500734885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:23.49942133 +0000 UTC m=+644.121729640" watchObservedRunningTime="2026-11-03 22:34:23.500734885 +0000 UTC m=+644.123043192" Nov 03 22:34:24.008925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:24.008885 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:24.021686 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:24.021661 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:24.405998 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:24.405971 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:34:24.500378 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:34:24.500341 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:34:28.366697 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:28.366650 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:34:32.413549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:32.413517 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:32.426431 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:32.426403 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:32.447711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:32.447684 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:34:49.427400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.427360 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh"] Nov 03 22:34:49.432414 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.432390 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.493240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.493204 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh"] Nov 03 22:34:49.530405 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.530372 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"openshift-service-ca.crt\"" Nov 03 22:34:49.530609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.530475 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"kube-root-ca.crt\"" Nov 03 22:34:49.530665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.530634 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"simulator-w4-18c03a-kserve-self-signed-certs\"" Nov 03 22:34:49.530665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.530654 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"default-dockercfg-6xfnv\"" Nov 03 22:34:49.561408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.561369 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-model-cache\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.561797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.561774 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tmp-dir\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.568492 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.568455 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-dshm\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.568938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.568911 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tls-certs\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.570277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.570252 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnkc\" (UniqueName: \"kubernetes.io/projected/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-kube-api-access-sbnkc\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.571347 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.571327 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-home\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673115 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673081 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tls-certs\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673136 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnkc\" (UniqueName: \"kubernetes.io/projected/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-kube-api-access-sbnkc\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673159 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-home\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673204 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-model-cache\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673234 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tmp-dir\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673524 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673353 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-dshm\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673618 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-home\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673637 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tmp-dir\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.673708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.673688 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-model-cache\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.675651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.675621 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-dshm\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.675849 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.675832 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tls-certs\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.730765 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.730693 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnkc\" (UniqueName: \"kubernetes.io/projected/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-kube-api-access-sbnkc\") pod \"simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:49.745262 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:49.745229 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:50.119363 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:50.119330 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh"] Nov 03 22:34:50.126247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:34:50.126217 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff43529e_2bec_4c7e_b8d4_a2c0b3d14a49.slice/crio-464c5198658eb74e266000e306a504f97350be7c35b066b4e43813334ca6f5a5 WatchSource:0}: Error finding container 464c5198658eb74e266000e306a504f97350be7c35b066b4e43813334ca6f5a5: Status 404 returned error can't find the container with id 464c5198658eb74e266000e306a504f97350be7c35b066b4e43813334ca6f5a5 Nov 03 22:34:50.504548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:50.504460 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:34:50.551327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:50.551291 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" event={"ID":"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49","Type":"ContainerStarted","Data":"1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2"} Nov 03 22:34:50.551327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:50.551333 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" event={"ID":"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49","Type":"ContainerStarted","Data":"464c5198658eb74e266000e306a504f97350be7c35b066b4e43813334ca6f5a5"} Nov 03 22:34:50.661598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:50.661533 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podStartSLOduration=1.661515254 podStartE2EDuration="1.661515254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:50.661011742 +0000 UTC m=+671.283320056" watchObservedRunningTime="2026-11-03 22:34:50.661515254 +0000 UTC m=+671.283823563" Nov 03 22:34:50.745790 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:50.745756 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:54.499913 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:34:54.499875 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:34:59.750606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:59.750567 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:34:59.769082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:34:59.769052 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:35:00.625143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:00.625114 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:35:07.933047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:07.933011 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:35:23.603435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.603396 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499"] Nov 03 22:35:23.606729 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.606709 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.660382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.660350 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ee74cb\"/\"kube-root-ca.crt\"" Nov 03 22:35:23.669513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.669482 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ee74cb\"/\"openshift-service-ca.crt\"" Nov 03 22:35:23.701926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.701888 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ee74cb\"/\"default-dockercfg-ptcqx\"" Nov 03 22:35:23.702103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.701984 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ee74cb\"/\"premium-w2-ee74cb-kserve-self-signed-certs\"" Nov 03 22:35:23.703962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.703941 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499"] Nov 03 22:35:23.746634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.746384 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-model-cache\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.750125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.749384 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-home\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.750125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.749468 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnk2\" (UniqueName: \"kubernetes.io/projected/08311447-fe46-43b3-aa87-86f1fa65136b-kube-api-access-8cnk2\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.750125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.749504 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08311447-fe46-43b3-aa87-86f1fa65136b-tls-certs\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.750125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.749537 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-tmp-dir\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.750125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.749621 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-dshm\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.850500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.850458 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-dshm\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.850709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.850550 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-model-cache\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.850781 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.850703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-home\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.850781 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.850770 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnk2\" (UniqueName: \"kubernetes.io/projected/08311447-fe46-43b3-aa87-86f1fa65136b-kube-api-access-8cnk2\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.850951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.850826 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08311447-fe46-43b3-aa87-86f1fa65136b-tls-certs\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.850951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.850872 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-tmp-dir\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.851117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.851092 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-model-cache\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.851245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.851092 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-home\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.851245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.851185 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-tmp-dir\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.853014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.852987 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-dshm\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.853462 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.853422 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08311447-fe46-43b3-aa87-86f1fa65136b-tls-certs\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.908421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.908391 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnk2\" (UniqueName: \"kubernetes.io/projected/08311447-fe46-43b3-aa87-86f1fa65136b-kube-api-access-8cnk2\") pod \"premium-w2-ee74cb-kserve-6c84789dd8-fj499\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:23.919254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:23.919226 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:24.324632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:24.324599 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499"] Nov 03 22:35:24.330989 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:35:24.330958 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08311447_fe46_43b3_aa87_86f1fa65136b.slice/crio-bbf1a7c8a3a6ffd2c965ed608540fe208bf9a3afa83f5ba6ff780efa1196c0af WatchSource:0}: Error finding container bbf1a7c8a3a6ffd2c965ed608540fe208bf9a3afa83f5ba6ff780efa1196c0af: Status 404 returned error can't find the container with id bbf1a7c8a3a6ffd2c965ed608540fe208bf9a3afa83f5ba6ff780efa1196c0af Nov 03 22:35:24.501101 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:35:24.501060 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:35:24.797878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:24.797827 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" event={"ID":"08311447-fe46-43b3-aa87-86f1fa65136b","Type":"ContainerStarted","Data":"3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb"} Nov 03 22:35:24.798388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:24.797886 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" event={"ID":"08311447-fe46-43b3-aa87-86f1fa65136b","Type":"ContainerStarted","Data":"bbf1a7c8a3a6ffd2c965ed608540fe208bf9a3afa83f5ba6ff780efa1196c0af"} Nov 03 22:35:24.907550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:24.907493 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podStartSLOduration=1.907480379 podStartE2EDuration="1.907480379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:24.906992248 +0000 UTC m=+705.529300558" watchObservedRunningTime="2026-11-03 22:35:24.907480379 +0000 UTC m=+705.529788686" Nov 03 22:35:24.920182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:24.920151 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:32.427921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:32.427881 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:35:33.919869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:33.919828 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:33.932567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:33.932535 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:34.869483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:34.869446 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:35:35.060693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.060657 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv"] Nov 03 22:35:35.066768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.066743 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.116889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.116854 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-961054\"/\"premium-w3-961054-kserve-self-signed-certs\"" Nov 03 22:35:35.119732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.119658 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv"] Nov 03 22:35:35.179394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.179346 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-model-cache\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.179394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.179386 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-dshm\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.179662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.179509 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qkf\" (UniqueName: \"kubernetes.io/projected/b266a56d-c69e-4051-94c8-6037b6b5d21e-kube-api-access-t2qkf\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.179662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.179568 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-home\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.179662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.179595 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-tmp-dir\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.179852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.179690 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b266a56d-c69e-4051-94c8-6037b6b5d21e-tls-certs\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281167 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qkf\" (UniqueName: \"kubernetes.io/projected/b266a56d-c69e-4051-94c8-6037b6b5d21e-kube-api-access-t2qkf\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281240 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-home\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281261 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-tmp-dir\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281281 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b266a56d-c69e-4051-94c8-6037b6b5d21e-tls-certs\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281349 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-model-cache\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281366 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-dshm\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281672 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-home\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281741 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-tmp-dir\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.281883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.281862 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-model-cache\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.283775 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.283744 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-dshm\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.284051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.284031 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b266a56d-c69e-4051-94c8-6037b6b5d21e-tls-certs\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.339067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.339031 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qkf\" (UniqueName: \"kubernetes.io/projected/b266a56d-c69e-4051-94c8-6037b6b5d21e-kube-api-access-t2qkf\") pod \"premium-w3-961054-kserve-d8c56d8cd-t2hsv\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.380193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.380107 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:35.787413 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.787380 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv"] Nov 03 22:35:35.794533 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:35:35.794501 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb266a56d_c69e_4051_94c8_6037b6b5d21e.slice/crio-a4be060de5d3cf76f49a463cddfc69e0f6cd083f4d0445415476d564b93907e0 WatchSource:0}: Error finding container a4be060de5d3cf76f49a463cddfc69e0f6cd083f4d0445415476d564b93907e0: Status 404 returned error can't find the container with id a4be060de5d3cf76f49a463cddfc69e0f6cd083f4d0445415476d564b93907e0 Nov 03 22:35:35.864291 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:35.864254 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" event={"ID":"b266a56d-c69e-4051-94c8-6037b6b5d21e","Type":"ContainerStarted","Data":"a4be060de5d3cf76f49a463cddfc69e0f6cd083f4d0445415476d564b93907e0"} Nov 03 22:35:36.871081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:36.871047 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" event={"ID":"b266a56d-c69e-4051-94c8-6037b6b5d21e","Type":"ContainerStarted","Data":"d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb"} Nov 03 22:35:36.982963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:36.982914 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podStartSLOduration=1.982900389 podStartE2EDuration="1.982900389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:36.980483474 +0000 UTC m=+717.602791795" watchObservedRunningTime="2026-11-03 22:35:36.982900389 +0000 UTC m=+717.605208743" Nov 03 22:35:37.380380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:37.380340 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:45.381206 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:45.381169 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:45.393659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:45.393625 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:45.931070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:45.931039 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:35:54.501216 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:35:54.501119 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:35:54.745568 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:35:54.745530 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:36:00.120088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.120051 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj"] Nov 03 22:36:00.124520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.124499 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.178291 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.178259 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj"] Nov 03 22:36:00.222003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.221973 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:36:00.222168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.222038 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:36:00.228769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.222526 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:36:00.228769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.222619 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:36:00.269153 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.269118 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzj78\" (UniqueName: \"kubernetes.io/projected/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-kube-api-access-dzj78\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.269346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.269173 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.269346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.269241 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.370996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.370901 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzj78\" (UniqueName: \"kubernetes.io/projected/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-kube-api-access-dzj78\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.370996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.370950 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.371232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.371071 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.371669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.371643 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.373395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.373375 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.428726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.428691 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzj78\" (UniqueName: \"kubernetes.io/projected/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-kube-api-access-dzj78\") pod \"maas-api-revoke-sub-4b44a677-cq5kj\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.437266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.437244 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:00.807935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:00.807901 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj"] Nov 03 22:36:00.814921 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:36:00.814889 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cd0ebc_7ecc_44f3_9dc0_b128c68e628b.slice/crio-8c9533708b0528827ddf2805d4f423791deeaa5a5f2816bc98b8caf3fdc2fc8c WatchSource:0}: Error finding container 8c9533708b0528827ddf2805d4f423791deeaa5a5f2816bc98b8caf3fdc2fc8c: Status 404 returned error can't find the container with id 8c9533708b0528827ddf2805d4f423791deeaa5a5f2816bc98b8caf3fdc2fc8c Nov 03 22:36:01.011878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:01.011839 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" event={"ID":"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b","Type":"ContainerStarted","Data":"8c9533708b0528827ddf2805d4f423791deeaa5a5f2816bc98b8caf3fdc2fc8c"} Nov 03 22:36:02.020269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:02.020236 5272 generic.go:356] "Generic (PLEG): container finished" podID="79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" containerID="0b6c889ca529c4893c5c707cf2eb2219dd542d8a962148b6595c68c9a6c84e08" exitCode=0 Nov 03 22:36:02.020665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:02.020282 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" event={"ID":"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b","Type":"ContainerDied","Data":"0b6c889ca529c4893c5c707cf2eb2219dd542d8a962148b6595c68c9a6c84e08"} Nov 03 22:36:03.170883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.170851 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:03.304295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.304249 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-openshift-ca\") pod \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " Nov 03 22:36:03.304496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.304369 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-kube-api-access-dzj78\" (UniqueName: \"kubernetes.io/projected/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-kube-api-access-dzj78\") pod \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " Nov 03 22:36:03.304496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.304444 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-xks-ca\") pod \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\" (UID: \"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b\") " Nov 03 22:36:03.304716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.304687 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-openshift-ca" pod "79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" (UID: "79cd0ebc-7ecc-44f3-9dc0-b128c68e628b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:03.304912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.304894 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:03.306583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.306552 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-kube-api-access-dzj78" pod "79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" (UID: "79cd0ebc-7ecc-44f3-9dc0-b128c68e628b"). InnerVolumeSpecName "kube-api-access-dzj78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:03.306883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.306858 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-xks-ca" pod "79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" (UID: "79cd0ebc-7ecc-44f3-9dc0-b128c68e628b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:03.406794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.406758 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzj78\" (UniqueName: \"kubernetes.io/projected/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-kube-api-access-dzj78\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:03.406794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:03.406787 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:04.032731 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:04.032698 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" Nov 03 22:36:04.032930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:04.032725 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj" event={"ID":"79cd0ebc-7ecc-44f3-9dc0-b128c68e628b","Type":"ContainerDied","Data":"8c9533708b0528827ddf2805d4f423791deeaa5a5f2816bc98b8caf3fdc2fc8c"} Nov 03 22:36:04.032930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:04.032794 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c9533708b0528827ddf2805d4f423791deeaa5a5f2816bc98b8caf3fdc2fc8c" Nov 03 22:36:17.695410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:17.695373 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:36:24.500369 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:36:24.500325 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:36:28.798354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.798320 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf"] Nov 03 22:36:28.798827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.798790 5272 cpu_manager.go:414] "containerMap: removing container" podUID="79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" containerName="revoke-keys" Nov 03 22:36:28.798827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.798816 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" containerName="revoke-keys" Nov 03 22:36:28.798920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.798910 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" containerName="revoke-keys" Nov 03 22:36:28.802190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.802170 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.853750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.853720 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:36:28.891397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.891369 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.891561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.891451 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.891561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.891508 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtn7v\" (UniqueName: \"kubernetes.io/projected/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-kube-api-access-mtn7v\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.900186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.900160 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:36:28.900186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.900171 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:36:28.900344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.900233 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:36:28.902289 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.902270 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf"] Nov 03 22:36:28.992326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.992294 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.992484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.992352 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.992484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.992376 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtn7v\" (UniqueName: \"kubernetes.io/projected/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-kube-api-access-mtn7v\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.993099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.993069 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:28.994768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:28.994746 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:29.013316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.013287 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2"] Nov 03 22:36:29.017388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.017369 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.050461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.050408 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtn7v\" (UniqueName: \"kubernetes.io/projected/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-kube-api-access-mtn7v\") pod \"maas-api-revoke-sub-4f7d47e7-kpcnf\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:29.072137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.072108 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2"] Nov 03 22:36:29.093390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.093349 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtlf\" (UniqueName: \"kubernetes.io/projected/2eee0603-f52a-44ad-9616-016d8157330d-kube-api-access-djtlf\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.093595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.093557 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.093668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.093600 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.115038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.115005 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:29.195345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.195245 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtlf\" (UniqueName: \"kubernetes.io/projected/2eee0603-f52a-44ad-9616-016d8157330d-kube-api-access-djtlf\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.195345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.195335 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.195548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.195355 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.196417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.196390 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.198454 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.198433 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.252787 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.252756 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtlf\" (UniqueName: \"kubernetes.io/projected/2eee0603-f52a-44ad-9616-016d8157330d-kube-api-access-djtlf\") pod \"maas-api-revoke-sub-c4fe534a-dcnn2\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.330479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.330401 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:29.493129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.493105 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf"] Nov 03 22:36:29.495154 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:36:29.495125 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb179e2_f3a5_4540_8fc7_fb7eb55f8b6a.slice/crio-35b1054a2d3fd40e40dfd3bfc46d05298f7e1d8b8d7d0311634ca9f70002fac7 WatchSource:0}: Error finding container 35b1054a2d3fd40e40dfd3bfc46d05298f7e1d8b8d7d0311634ca9f70002fac7: Status 404 returned error can't find the container with id 35b1054a2d3fd40e40dfd3bfc46d05298f7e1d8b8d7d0311634ca9f70002fac7 Nov 03 22:36:29.702169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:29.702130 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2"] Nov 03 22:36:29.708772 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:36:29.708744 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eee0603_f52a_44ad_9616_016d8157330d.slice/crio-364925c761cf86918d025e58ce48aa92345b601e51fdaa8e6aef95dda7763724 WatchSource:0}: Error finding container 364925c761cf86918d025e58ce48aa92345b601e51fdaa8e6aef95dda7763724: Status 404 returned error can't find the container with id 364925c761cf86918d025e58ce48aa92345b601e51fdaa8e6aef95dda7763724 Nov 03 22:36:30.177419 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:30.177374 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" event={"ID":"2eee0603-f52a-44ad-9616-016d8157330d","Type":"ContainerStarted","Data":"6aa685ee446297b038e8a57854c596fd180e5ee53687412c32e1528a23e4985e"} Nov 03 22:36:30.177863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:30.177482 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" event={"ID":"2eee0603-f52a-44ad-9616-016d8157330d","Type":"ContainerStarted","Data":"364925c761cf86918d025e58ce48aa92345b601e51fdaa8e6aef95dda7763724"} Nov 03 22:36:30.179047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:30.179020 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" event={"ID":"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a","Type":"ContainerStarted","Data":"43d1efb0b14680eb3f0a0632459cf156ea27ff87eca450c7a1ffd68d2a94d413"} Nov 03 22:36:30.179137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:30.179069 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" event={"ID":"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a","Type":"ContainerStarted","Data":"35b1054a2d3fd40e40dfd3bfc46d05298f7e1d8b8d7d0311634ca9f70002fac7"} Nov 03 22:36:30.287718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:30.287661 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" podStartSLOduration=2.28764753 podStartE2EDuration="2.28764753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:30.28523665 +0000 UTC m=+770.907544958" watchObservedRunningTime="2026-11-03 22:36:30.28764753 +0000 UTC m=+770.909955873" Nov 03 22:36:30.394814 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:30.394752 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" podStartSLOduration=2.394734845 podStartE2EDuration="2.394734845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:30.393771511 +0000 UTC m=+771.016079821" watchObservedRunningTime="2026-11-03 22:36:30.394734845 +0000 UTC m=+771.017043155" Nov 03 22:36:31.185872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:31.185836 5272 generic.go:356] "Generic (PLEG): container finished" podID="2eee0603-f52a-44ad-9616-016d8157330d" containerID="6aa685ee446297b038e8a57854c596fd180e5ee53687412c32e1528a23e4985e" exitCode=0 Nov 03 22:36:31.186298 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:31.185924 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" event={"ID":"2eee0603-f52a-44ad-9616-016d8157330d","Type":"ContainerDied","Data":"6aa685ee446297b038e8a57854c596fd180e5ee53687412c32e1528a23e4985e"} Nov 03 22:36:31.187553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:31.187527 5272 generic.go:356] "Generic (PLEG): container finished" podID="bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" containerID="43d1efb0b14680eb3f0a0632459cf156ea27ff87eca450c7a1ffd68d2a94d413" exitCode=0 Nov 03 22:36:31.187663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:31.187628 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" event={"ID":"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a","Type":"ContainerDied","Data":"43d1efb0b14680eb3f0a0632459cf156ea27ff87eca450c7a1ffd68d2a94d413"} Nov 03 22:36:32.340437 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.340403 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:32.371171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.371146 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:32.439648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.439605 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-kube-api-access-mtn7v\" (UniqueName: \"kubernetes.io/projected/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-kube-api-access-mtn7v\") pod \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " Nov 03 22:36:32.439882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.439721 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eee0603-f52a-44ad-9616-016d8157330d-kube-api-access-djtlf\" (UniqueName: \"kubernetes.io/projected/2eee0603-f52a-44ad-9616-016d8157330d-kube-api-access-djtlf\") pod \"2eee0603-f52a-44ad-9616-016d8157330d\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " Nov 03 22:36:32.439882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.439784 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-xks-ca\") pod \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " Nov 03 22:36:32.439882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.439850 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-openshift-ca\") pod \"2eee0603-f52a-44ad-9616-016d8157330d\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " Nov 03 22:36:32.440057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.439902 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-xks-ca\") pod \"2eee0603-f52a-44ad-9616-016d8157330d\" (UID: \"2eee0603-f52a-44ad-9616-016d8157330d\") " Nov 03 22:36:32.440057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.439955 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-openshift-ca\") pod \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\" (UID: \"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a\") " Nov 03 22:36:32.440701 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.440663 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-openshift-ca" pod "bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" (UID: "bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:32.441851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.441824 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-openshift-ca" pod "2eee0603-f52a-44ad-9616-016d8157330d" (UID: "2eee0603-f52a-44ad-9616-016d8157330d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:32.443702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.443637 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eee0603-f52a-44ad-9616-016d8157330d-kube-api-access-djtlf" pod "2eee0603-f52a-44ad-9616-016d8157330d" (UID: "2eee0603-f52a-44ad-9616-016d8157330d"). InnerVolumeSpecName "kube-api-access-djtlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:32.444384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.444345 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-kube-api-access-mtn7v" pod "bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" (UID: "bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a"). InnerVolumeSpecName "kube-api-access-mtn7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:32.444384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.444355 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-xks-ca" pod "2eee0603-f52a-44ad-9616-016d8157330d" (UID: "2eee0603-f52a-44ad-9616-016d8157330d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:32.445531 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.445510 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-xks-ca" pod "bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" (UID: "bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:32.541874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.541833 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtlf\" (UniqueName: \"kubernetes.io/projected/2eee0603-f52a-44ad-9616-016d8157330d-kube-api-access-djtlf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:32.541874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.541871 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:32.542102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.541886 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:32.542102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.541901 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eee0603-f52a-44ad-9616-016d8157330d-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:32.542102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.541915 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:32.542102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:32.541930 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtn7v\" (UniqueName: \"kubernetes.io/projected/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a-kube-api-access-mtn7v\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:33.198991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:33.198953 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" Nov 03 22:36:33.198991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:33.198980 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf" event={"ID":"bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a","Type":"ContainerDied","Data":"35b1054a2d3fd40e40dfd3bfc46d05298f7e1d8b8d7d0311634ca9f70002fac7"} Nov 03 22:36:33.199265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:33.199032 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b1054a2d3fd40e40dfd3bfc46d05298f7e1d8b8d7d0311634ca9f70002fac7" Nov 03 22:36:33.200671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:33.200638 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" event={"ID":"2eee0603-f52a-44ad-9616-016d8157330d","Type":"ContainerDied","Data":"364925c761cf86918d025e58ce48aa92345b601e51fdaa8e6aef95dda7763724"} Nov 03 22:36:33.200671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:33.200665 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2" Nov 03 22:36:33.200868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:33.200690 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="364925c761cf86918d025e58ce48aa92345b601e51fdaa8e6aef95dda7763724" Nov 03 22:36:44.468452 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.468417 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf"] Nov 03 22:36:44.468960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.468938 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2eee0603-f52a-44ad-9616-016d8157330d" containerName="revoke-keys" Nov 03 22:36:44.468960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.468950 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eee0603-f52a-44ad-9616-016d8157330d" containerName="revoke-keys" Nov 03 22:36:44.469037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.468976 5272 cpu_manager.go:414] "containerMap: removing container" podUID="bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" containerName="revoke-keys" Nov 03 22:36:44.469037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.468981 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" containerName="revoke-keys" Nov 03 22:36:44.469099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.469069 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" containerName="revoke-keys" Nov 03 22:36:44.469099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.469084 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eee0603-f52a-44ad-9616-016d8157330d" containerName="revoke-keys" Nov 03 22:36:44.474101 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.474070 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.524526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.524494 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:36:44.570662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.570628 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:36:44.570871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.570857 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:36:44.570933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.570881 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:36:44.573293 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.573272 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf"] Nov 03 22:36:44.575279 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.575255 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgt9\" (UniqueName: \"kubernetes.io/projected/4fa643b7-29d3-4eac-a45d-15bc0b90555a-kube-api-access-vwgt9\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.575471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.575452 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.575537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.575521 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.677116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.677071 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.677323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.677136 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgt9\" (UniqueName: \"kubernetes.io/projected/4fa643b7-29d3-4eac-a45d-15bc0b90555a-kube-api-access-vwgt9\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.677323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.677304 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.677740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.677715 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.679722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.679697 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.734986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.734900 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgt9\" (UniqueName: \"kubernetes.io/projected/4fa643b7-29d3-4eac-a45d-15bc0b90555a-kube-api-access-vwgt9\") pod \"maas-api-revoke-sub-0c9ef24b-pd7mf\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:44.788228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:44.788183 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:45.161234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:45.161199 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf"] Nov 03 22:36:45.167743 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:36:45.167715 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa643b7_29d3_4eac_a45d_15bc0b90555a.slice/crio-44bd027e4680c6eac28e420587ef803a25f7085b22e7606eee09d97d5fea997b WatchSource:0}: Error finding container 44bd027e4680c6eac28e420587ef803a25f7085b22e7606eee09d97d5fea997b: Status 404 returned error can't find the container with id 44bd027e4680c6eac28e420587ef803a25f7085b22e7606eee09d97d5fea997b Nov 03 22:36:45.276577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:45.276543 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" event={"ID":"4fa643b7-29d3-4eac-a45d-15bc0b90555a","Type":"ContainerStarted","Data":"44bd027e4680c6eac28e420587ef803a25f7085b22e7606eee09d97d5fea997b"} Nov 03 22:36:46.283859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:46.283824 5272 generic.go:356] "Generic (PLEG): container finished" podID="4fa643b7-29d3-4eac-a45d-15bc0b90555a" containerID="90c1d0737905d6b655b7191548d008881f766c77a2218cc4e7e331b22be8f572" exitCode=0 Nov 03 22:36:46.284274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:46.283909 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" event={"ID":"4fa643b7-29d3-4eac-a45d-15bc0b90555a","Type":"ContainerDied","Data":"90c1d0737905d6b655b7191548d008881f766c77a2218cc4e7e331b22be8f572"} Nov 03 22:36:47.431215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.431191 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:47.608006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.607920 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-xks-ca\") pod \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " Nov 03 22:36:47.608154 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.608064 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-openshift-ca\") pod \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " Nov 03 22:36:47.608195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.608166 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa643b7-29d3-4eac-a45d-15bc0b90555a-kube-api-access-vwgt9\" (UniqueName: \"kubernetes.io/projected/4fa643b7-29d3-4eac-a45d-15bc0b90555a-kube-api-access-vwgt9\") pod \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\" (UID: \"4fa643b7-29d3-4eac-a45d-15bc0b90555a\") " Nov 03 22:36:47.608428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.608403 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-openshift-ca" pod "4fa643b7-29d3-4eac-a45d-15bc0b90555a" (UID: "4fa643b7-29d3-4eac-a45d-15bc0b90555a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:47.608595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.608579 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:47.610199 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.610171 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa643b7-29d3-4eac-a45d-15bc0b90555a-kube-api-access-vwgt9" pod "4fa643b7-29d3-4eac-a45d-15bc0b90555a" (UID: "4fa643b7-29d3-4eac-a45d-15bc0b90555a"). InnerVolumeSpecName "kube-api-access-vwgt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:47.610290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.610199 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-xks-ca" pod "4fa643b7-29d3-4eac-a45d-15bc0b90555a" (UID: "4fa643b7-29d3-4eac-a45d-15bc0b90555a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:47.709964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.709921 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwgt9\" (UniqueName: \"kubernetes.io/projected/4fa643b7-29d3-4eac-a45d-15bc0b90555a-kube-api-access-vwgt9\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:47.709964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:47.709961 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa643b7-29d3-4eac-a45d-15bc0b90555a-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:48.296918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:48.296880 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" event={"ID":"4fa643b7-29d3-4eac-a45d-15bc0b90555a","Type":"ContainerDied","Data":"44bd027e4680c6eac28e420587ef803a25f7085b22e7606eee09d97d5fea997b"} Nov 03 22:36:48.297137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:48.296957 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44bd027e4680c6eac28e420587ef803a25f7085b22e7606eee09d97d5fea997b" Nov 03 22:36:48.297137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:48.297013 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf" Nov 03 22:36:49.606432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.606396 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj"] Nov 03 22:36:49.606920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.606905 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4fa643b7-29d3-4eac-a45d-15bc0b90555a" containerName="revoke-keys" Nov 03 22:36:49.606981 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.606920 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa643b7-29d3-4eac-a45d-15bc0b90555a" containerName="revoke-keys" Nov 03 22:36:49.607032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.607023 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa643b7-29d3-4eac-a45d-15bc0b90555a" containerName="revoke-keys" Nov 03 22:36:49.610551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.610529 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.661938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.661907 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:36:49.662112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.661955 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:36:49.708386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.708351 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:36:49.708386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.708358 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:36:49.710659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.710633 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj"] Nov 03 22:36:49.731536 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.731509 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zch47\" (UniqueName: \"kubernetes.io/projected/df235505-0f1b-4b05-9d8a-957d00215ef8-kube-api-access-zch47\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.731654 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.731549 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.731654 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.731569 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.833193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.833164 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zch47\" (UniqueName: \"kubernetes.io/projected/df235505-0f1b-4b05-9d8a-957d00215ef8-kube-api-access-zch47\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.833364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.833206 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.833364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.833226 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.833789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.833769 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.835541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.835519 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.890588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.890508 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zch47\" (UniqueName: \"kubernetes.io/projected/df235505-0f1b-4b05-9d8a-957d00215ef8-kube-api-access-zch47\") pod \"maas-api-revoke-sub-c07e5fb8-flvhj\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:49.926414 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:49.926384 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:50.299491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:50.299451 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj"] Nov 03 22:36:50.306504 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:36:50.306472 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf235505_0f1b_4b05_9d8a_957d00215ef8.slice/crio-1e6e8d74869653bb1e1f606fd25b214dfe580561d8b2bded859ca678685af3b0 WatchSource:0}: Error finding container 1e6e8d74869653bb1e1f606fd25b214dfe580561d8b2bded859ca678685af3b0: Status 404 returned error can't find the container with id 1e6e8d74869653bb1e1f606fd25b214dfe580561d8b2bded859ca678685af3b0 Nov 03 22:36:51.315354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:51.315315 5272 generic.go:356] "Generic (PLEG): container finished" podID="df235505-0f1b-4b05-9d8a-957d00215ef8" containerID="97eacd6994a02ac5df88df2e97faa92a169967e929ea00587214cea5099be4de" exitCode=0 Nov 03 22:36:51.315908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:51.315454 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" event={"ID":"df235505-0f1b-4b05-9d8a-957d00215ef8","Type":"ContainerDied","Data":"97eacd6994a02ac5df88df2e97faa92a169967e929ea00587214cea5099be4de"} Nov 03 22:36:51.315908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:51.315505 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" event={"ID":"df235505-0f1b-4b05-9d8a-957d00215ef8","Type":"ContainerStarted","Data":"1e6e8d74869653bb1e1f606fd25b214dfe580561d8b2bded859ca678685af3b0"} Nov 03 22:36:52.465041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.465017 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:52.562432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.562398 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-xks-ca\") pod \"df235505-0f1b-4b05-9d8a-957d00215ef8\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " Nov 03 22:36:52.564442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.564419 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-xks-ca" pod "df235505-0f1b-4b05-9d8a-957d00215ef8" (UID: "df235505-0f1b-4b05-9d8a-957d00215ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:52.663257 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.663161 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-openshift-ca\") pod \"df235505-0f1b-4b05-9d8a-957d00215ef8\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " Nov 03 22:36:52.663257 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.663246 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df235505-0f1b-4b05-9d8a-957d00215ef8-kube-api-access-zch47\" (UniqueName: \"kubernetes.io/projected/df235505-0f1b-4b05-9d8a-957d00215ef8-kube-api-access-zch47\") pod \"df235505-0f1b-4b05-9d8a-957d00215ef8\" (UID: \"df235505-0f1b-4b05-9d8a-957d00215ef8\") " Nov 03 22:36:52.663542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.663518 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-openshift-ca" pod "df235505-0f1b-4b05-9d8a-957d00215ef8" (UID: "df235505-0f1b-4b05-9d8a-957d00215ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:52.663662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.663644 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:52.663710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.663668 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df235505-0f1b-4b05-9d8a-957d00215ef8-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:52.665460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.665435 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df235505-0f1b-4b05-9d8a-957d00215ef8-kube-api-access-zch47" pod "df235505-0f1b-4b05-9d8a-957d00215ef8" (UID: "df235505-0f1b-4b05-9d8a-957d00215ef8"). InnerVolumeSpecName "kube-api-access-zch47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:52.764321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:52.764280 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zch47\" (UniqueName: \"kubernetes.io/projected/df235505-0f1b-4b05-9d8a-957d00215ef8-kube-api-access-zch47\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:53.327872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:53.327835 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" Nov 03 22:36:53.327872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:53.327864 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj" event={"ID":"df235505-0f1b-4b05-9d8a-957d00215ef8","Type":"ContainerDied","Data":"1e6e8d74869653bb1e1f606fd25b214dfe580561d8b2bded859ca678685af3b0"} Nov 03 22:36:53.328090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:36:53.327918 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e6e8d74869653bb1e1f606fd25b214dfe580561d8b2bded859ca678685af3b0" Nov 03 22:36:54.500644 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:36:54.500605 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:37:00.313303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.313265 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms"] Nov 03 22:37:00.313854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.313837 5272 cpu_manager.go:414] "containerMap: removing container" podUID="df235505-0f1b-4b05-9d8a-957d00215ef8" containerName="revoke-keys" Nov 03 22:37:00.313907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.313855 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df235505-0f1b-4b05-9d8a-957d00215ef8" containerName="revoke-keys" Nov 03 22:37:00.313950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.313939 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="df235505-0f1b-4b05-9d8a-957d00215ef8" containerName="revoke-keys" Nov 03 22:37:00.317233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.317209 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.347644 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.347612 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.347786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.347652 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zgr\" (UniqueName: \"kubernetes.io/projected/358d085e-31a0-4a0a-9b80-fbbea541fed5-kube-api-access-89zgr\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.347786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.347713 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.390079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.390047 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:00.437051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.437011 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:00.439128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.439106 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms"] Nov 03 22:37:00.448899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.448873 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.449035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.448920 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zgr\" (UniqueName: \"kubernetes.io/projected/358d085e-31a0-4a0a-9b80-fbbea541fed5-kube-api-access-89zgr\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.449035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.448972 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.451435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.451407 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.466648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.466621 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:00.470182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.470161 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.480195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.480173 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:37:00.518782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.518755 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zgr\" (UniqueName: \"kubernetes.io/projected/358d085e-31a0-4a0a-9b80-fbbea541fed5-kube-api-access-89zgr\") pod \"maas-api-revoke-sub-d7390395-cjwms\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:00.630396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:00.630309 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:01.004048 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:01.003958 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms"] Nov 03 22:37:01.010194 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:01.010165 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358d085e_31a0_4a0a_9b80_fbbea541fed5.slice/crio-34c2c979cd70ec3ed6b3dae8a26a6d379b20d4c2e4b0a4f91915eac207610f70 WatchSource:0}: Error finding container 34c2c979cd70ec3ed6b3dae8a26a6d379b20d4c2e4b0a4f91915eac207610f70: Status 404 returned error can't find the container with id 34c2c979cd70ec3ed6b3dae8a26a6d379b20d4c2e4b0a4f91915eac207610f70 Nov 03 22:37:01.387438 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:01.387380 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" event={"ID":"358d085e-31a0-4a0a-9b80-fbbea541fed5","Type":"ContainerStarted","Data":"7810407d7ce6fa2464fefa8aec24d6f9ae7351375f5c837abddc8273f7bd4e9f"} Nov 03 22:37:01.387962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:01.387470 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" event={"ID":"358d085e-31a0-4a0a-9b80-fbbea541fed5","Type":"ContainerStarted","Data":"34c2c979cd70ec3ed6b3dae8a26a6d379b20d4c2e4b0a4f91915eac207610f70"} Nov 03 22:37:01.499684 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:01.499631 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" podStartSLOduration=1.4996179029999999 podStartE2EDuration="1.499617903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:01.497305512 +0000 UTC m=+802.119613819" watchObservedRunningTime="2026-11-03 22:37:01.499617903 +0000 UTC m=+802.121926210" Nov 03 22:37:01.551779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:01.551746 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:37:02.175234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.175138 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7"] Nov 03 22:37:02.179360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.179336 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.228846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.228322 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ee74cb-openshift-default-dockercfg-zszk2\"" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268406 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268489 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268597 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268625 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxgt\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-kube-api-access-8zxgt\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268654 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268688 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faef726-470f-4c18-99cd-4569fdb2d878-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268725 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268869 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.272789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.268921 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faef726-470f-4c18-99cd-4569fdb2d878-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.302844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.301859 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7"] Nov 03 22:37:02.369490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369447 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faef726-470f-4c18-99cd-4569fdb2d878-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.369698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369499 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.369698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369597 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.369698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369640 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faef726-470f-4c18-99cd-4569fdb2d878-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.369698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369675 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369701 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369790 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369849 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxgt\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-kube-api-access-8zxgt\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.369877 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.370035 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.370060 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.370272 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.370577 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faef726-470f-4c18-99cd-4569fdb2d878-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.370749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.370729 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.372085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.372062 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faef726-470f-4c18-99cd-4569fdb2d878-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.372617 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.372592 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.395250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.395210 5272 generic.go:356] "Generic (PLEG): container finished" podID="358d085e-31a0-4a0a-9b80-fbbea541fed5" containerID="7810407d7ce6fa2464fefa8aec24d6f9ae7351375f5c837abddc8273f7bd4e9f" exitCode=0 Nov 03 22:37:02.395710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.395301 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" event={"ID":"358d085e-31a0-4a0a-9b80-fbbea541fed5","Type":"ContainerDied","Data":"7810407d7ce6fa2464fefa8aec24d6f9ae7351375f5c837abddc8273f7bd4e9f"} Nov 03 22:37:02.427864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.427769 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.428007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.427963 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxgt\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-kube-api-access-8zxgt\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.493830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.493383 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:02.972314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.972266 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7"] Nov 03 22:37:02.983332 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:02.983296 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7faef726_470f_4c18_99cd_4569fdb2d878.slice/crio-3013663d91807d9ffe3f6bb56fe74932e9cff88e4b0222e00a07ea8e615547e6 WatchSource:0}: Error finding container 3013663d91807d9ffe3f6bb56fe74932e9cff88e4b0222e00a07ea8e615547e6: Status 404 returned error can't find the container with id 3013663d91807d9ffe3f6bb56fe74932e9cff88e4b0222e00a07ea8e615547e6 Nov 03 22:37:02.986177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.985958 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:02.986177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.986058 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:02.986177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:02.986115 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:03.408567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.408514 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" event={"ID":"7faef726-470f-4c18-99cd-4569fdb2d878","Type":"ContainerStarted","Data":"6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520"} Nov 03 22:37:03.409121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.408596 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" event={"ID":"7faef726-470f-4c18-99cd-4569fdb2d878","Type":"ContainerStarted","Data":"3013663d91807d9ffe3f6bb56fe74932e9cff88e4b0222e00a07ea8e615547e6"} Nov 03 22:37:03.494988 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.494938 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:03.524837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.524767 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podStartSLOduration=1.5247519459999999 podStartE2EDuration="1.524751946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:03.521909697 +0000 UTC m=+804.144218005" watchObservedRunningTime="2026-11-03 22:37:03.524751946 +0000 UTC m=+804.147060253" Nov 03 22:37:03.596219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.595872 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:03.606932 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.606266 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6"] Nov 03 22:37:03.606932 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.606521 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" containerID="cri-o://58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada" gracePeriod=30 Nov 03 22:37:03.616846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.615850 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql"] Nov 03 22:37:03.616846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.616069 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" containerID="cri-o://0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c" gracePeriod=30 Nov 03 22:37:03.689746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.689642 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358d085e-31a0-4a0a-9b80-fbbea541fed5-kube-api-access-89zgr\" (UniqueName: \"kubernetes.io/projected/358d085e-31a0-4a0a-9b80-fbbea541fed5-kube-api-access-89zgr\") pod \"358d085e-31a0-4a0a-9b80-fbbea541fed5\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " Nov 03 22:37:03.689746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.689715 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-openshift-ca\") pod \"358d085e-31a0-4a0a-9b80-fbbea541fed5\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " Nov 03 22:37:03.690000 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.689753 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-xks-ca\") pod \"358d085e-31a0-4a0a-9b80-fbbea541fed5\" (UID: \"358d085e-31a0-4a0a-9b80-fbbea541fed5\") " Nov 03 22:37:03.690206 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.690174 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-openshift-ca" pod "358d085e-31a0-4a0a-9b80-fbbea541fed5" (UID: "358d085e-31a0-4a0a-9b80-fbbea541fed5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:03.691885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.691860 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358d085e-31a0-4a0a-9b80-fbbea541fed5-kube-api-access-89zgr" pod "358d085e-31a0-4a0a-9b80-fbbea541fed5" (UID: "358d085e-31a0-4a0a-9b80-fbbea541fed5"). InnerVolumeSpecName "kube-api-access-89zgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:03.692588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.692567 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-xks-ca" pod "358d085e-31a0-4a0a-9b80-fbbea541fed5" (UID: "358d085e-31a0-4a0a-9b80-fbbea541fed5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:03.791519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.791471 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:03.791519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.791514 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89zgr\" (UniqueName: \"kubernetes.io/projected/358d085e-31a0-4a0a-9b80-fbbea541fed5-kube-api-access-89zgr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:03.791519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:03.791524 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358d085e-31a0-4a0a-9b80-fbbea541fed5-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:04.416691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:04.416657 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" Nov 03 22:37:04.417148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:04.416692 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms" event={"ID":"358d085e-31a0-4a0a-9b80-fbbea541fed5","Type":"ContainerDied","Data":"34c2c979cd70ec3ed6b3dae8a26a6d379b20d4c2e4b0a4f91915eac207610f70"} Nov 03 22:37:04.417148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:04.416748 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34c2c979cd70ec3ed6b3dae8a26a6d379b20d4c2e4b0a4f91915eac207610f70" Nov 03 22:37:04.495851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:04.495815 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:37:04.496034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:04.495873 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 22:37:05.493852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:05.493819 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:37:05.494236 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:05.493882 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 22:37:06.494358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:06.494324 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:37:06.494758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:06.494387 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 22:37:07.494826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:07.494441 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:37:07.494826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:07.494517 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 22:37:08.494664 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:08.494624 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:37:08.494881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:08.494697 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 22:37:08.955379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:08.955006 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:37:09.096500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096416 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-data\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096457 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-podinfo\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096482 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istiod-ca-cert\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096525 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-envoy\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096553 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-token\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-token\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096594 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-certs\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096643 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-kube-api-access-f48rg\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-kube-api-access-f48rg\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096732 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-socket\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.096995 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096747 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-data" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.096995 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.096882 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-credential-socket\") pod \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\" (UID: \"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9\") " Nov 03 22:37:09.097100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097001 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-socket" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.097158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097109 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-certs" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.097158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097041 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istiod-ca-cert" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:09.097308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097286 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-credential-socket" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.097415 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097389 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.097415 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097414 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.097612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097423 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.097612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097432 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.097612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.097444 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.098837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.098781 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-podinfo" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:09.099464 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.099441 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-token" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:09.099532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.099517 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-kube-api-access-f48rg" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "kube-api-access-f48rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:09.099599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.099580 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-envoy" pod "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" (UID: "85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.198839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.198482 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.198839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.198522 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.198839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.198536 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.198839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.198551 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f48rg\" (UniqueName: \"kubernetes.io/projected/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9-kube-api-access-f48rg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.269520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.269495 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:37:09.400789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.400692 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-socket\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.400789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.400772 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-envoy\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.400830 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-credential-socket\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.400871 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-data\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.400923 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-token\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-token\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.400998 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istiod-ca-cert\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401020 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-socket" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.401219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401028 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-kube-api-access-svq8c\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-kube-api-access-svq8c\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401177 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-certs\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401241 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-podinfo\") pod \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\" (UID: \"8a26eabd-5f49-4d26-9937-aa599ad02fd8\") " Nov 03 22:37:09.401425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401405 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-credential-socket" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.401742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401722 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.401862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401749 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.401926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.401869 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-data" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.402066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.402045 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-certs" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.402248 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.402215 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istiod-ca-cert" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:09.403579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.403552 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-envoy" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:09.404367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.404342 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-kube-api-access-svq8c" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "kube-api-access-svq8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:09.404732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.404704 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-token" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:09.404825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.404769 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-podinfo" pod "8a26eabd-5f49-4d26-9937-aa599ad02fd8" (UID: "8a26eabd-5f49-4d26-9937-aa599ad02fd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:09.449859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.449818 5272 generic.go:356] "Generic (PLEG): container finished" podID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerID="58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada" exitCode=0 Nov 03 22:37:09.450034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.449885 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" Nov 03 22:37:09.450034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.449887 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" event={"ID":"8a26eabd-5f49-4d26-9937-aa599ad02fd8","Type":"ContainerDied","Data":"58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada"} Nov 03 22:37:09.450034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.449975 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6" event={"ID":"8a26eabd-5f49-4d26-9937-aa599ad02fd8","Type":"ContainerDied","Data":"9f16cdad15c18a8b39a1803414e40664c1bf4fb0ec6551533b8daf4f61a14a67"} Nov 03 22:37:09.450034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.450007 5272 scope.go:122] "RemoveContainer" containerID="58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada" Nov 03 22:37:09.451664 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.451639 5272 generic.go:356] "Generic (PLEG): container finished" podID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerID="0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c" exitCode=0 Nov 03 22:37:09.451763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.451693 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" Nov 03 22:37:09.451763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.451704 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" event={"ID":"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9","Type":"ContainerDied","Data":"0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c"} Nov 03 22:37:09.451763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.451743 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql" event={"ID":"85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9","Type":"ContainerDied","Data":"4b0b3a6509a22bbc97718fb9b3663d514796b348cb2cbef8313a21b667e9ec78"} Nov 03 22:37:09.461377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.461359 5272 scope.go:122] "RemoveContainer" containerID="58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada" Nov 03 22:37:09.461770 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:37:09.461742 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada\": container with ID starting with 58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada not found: ID does not exist" containerID="58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada" Nov 03 22:37:09.461873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.461781 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada"} err="failed to get container status \"58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada\": rpc error: code = NotFound desc = could not find container \"58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada\": container with ID starting with 58967604e69d6a6a807643fef8a7dca713a51b4d8f81a47aa515ea3020883ada not found: ID does not exist" Nov 03 22:37:09.461873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.461827 5272 scope.go:122] "RemoveContainer" containerID="0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c" Nov 03 22:37:09.470648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.470628 5272 scope.go:122] "RemoveContainer" containerID="0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c" Nov 03 22:37:09.470930 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:37:09.470909 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c\": container with ID starting with 0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c not found: ID does not exist" containerID="0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c" Nov 03 22:37:09.471026 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.470935 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c"} err="failed to get container status \"0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c\": rpc error: code = NotFound desc = could not find container \"0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c\": container with ID starting with 0c22fd18b9c00231a818f2058feeaa8fc36089bc7f0d8a75471dc47031f9578c not found: ID does not exist" Nov 03 22:37:09.496522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.496494 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 22:37:09.497041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.496543 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 22:37:09.502758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.502731 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.502758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.502755 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.502975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.502766 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.502975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.502777 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.502975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.502786 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a26eabd-5f49-4d26-9937-aa599ad02fd8-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.502975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.502796 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svq8c\" (UniqueName: \"kubernetes.io/projected/8a26eabd-5f49-4d26-9937-aa599ad02fd8-kube-api-access-svq8c\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.502975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.502824 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a26eabd-5f49-4d26-9937-aa599ad02fd8-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:09.610394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.610357 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql"] Nov 03 22:37:09.614232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.614187 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8mmql"] Nov 03 22:37:09.726882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.722377 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6"] Nov 03 22:37:09.726882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:09.726845 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n2sn6"] Nov 03 22:37:10.498567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:10.498536 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:10.498982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:10.498815 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:10.499432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:10.499416 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:37:11.499166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:11.499129 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" path="/var/lib/kubelet/pods/85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9/volumes" Nov 03 22:37:11.499551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:11.499543 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" path="/var/lib/kubelet/pods/8a26eabd-5f49-4d26-9937-aa599ad02fd8/volumes" Nov 03 22:37:17.375159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375127 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb"] Nov 03 22:37:17.375655 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375610 5272 cpu_manager.go:414] "containerMap: removing container" podUID="358d085e-31a0-4a0a-9b80-fbbea541fed5" containerName="revoke-keys" Nov 03 22:37:17.375655 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375621 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358d085e-31a0-4a0a-9b80-fbbea541fed5" containerName="revoke-keys" Nov 03 22:37:17.375655 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375631 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" Nov 03 22:37:17.375655 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375637 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" Nov 03 22:37:17.375824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375662 5272 cpu_manager.go:414] "containerMap: removing container" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" Nov 03 22:37:17.375824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375667 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" Nov 03 22:37:17.375824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375735 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b097f6-6a43-4fc6-9a0c-e7727aa3b2e9" containerName="istio-proxy" Nov 03 22:37:17.375824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375746 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="358d085e-31a0-4a0a-9b80-fbbea541fed5" containerName="revoke-keys" Nov 03 22:37:17.375824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.375752 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a26eabd-5f49-4d26-9937-aa599ad02fd8" containerName="istio-proxy" Nov 03 22:37:17.380567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.380543 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.433784 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.433745 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb"] Nov 03 22:37:17.477209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.477183 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:17.477209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.477197 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:17.477423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.477229 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:37:17.477423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.477276 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:17.485750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.485725 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.485842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.485776 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbw9\" (UniqueName: \"kubernetes.io/projected/ed7755c6-a357-4cf8-b40c-70057cceaa34-kube-api-access-wzbw9\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.485991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.485973 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.586590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.586556 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.586755 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.586635 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.586755 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.586668 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbw9\" (UniqueName: \"kubernetes.io/projected/ed7755c6-a357-4cf8-b40c-70057cceaa34-kube-api-access-wzbw9\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.587332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.587306 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.588983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.588959 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.644274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.644203 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbw9\" (UniqueName: \"kubernetes.io/projected/ed7755c6-a357-4cf8-b40c-70057cceaa34-kube-api-access-wzbw9\") pod \"maas-api-revoke-sub-8eb365bd-7kqxb\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:17.724007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:17.723971 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:18.106508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:18.106469 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb"] Nov 03 22:37:18.112829 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:18.112783 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7755c6_a357_4cf8_b40c_70057cceaa34.slice/crio-65a23352f000ee4059a52fdc0bf7866d9435ea14e8d820ff9c4e9a267390525e WatchSource:0}: Error finding container 65a23352f000ee4059a52fdc0bf7866d9435ea14e8d820ff9c4e9a267390525e: Status 404 returned error can't find the container with id 65a23352f000ee4059a52fdc0bf7866d9435ea14e8d820ff9c4e9a267390525e Nov 03 22:37:18.509964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:18.509872 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" event={"ID":"ed7755c6-a357-4cf8-b40c-70057cceaa34","Type":"ContainerStarted","Data":"e3637455cb7b94d1386f1112888e1dc334d3c815682473f6310ce6dabf7d2c17"} Nov 03 22:37:18.509964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:18.509945 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" event={"ID":"ed7755c6-a357-4cf8-b40c-70057cceaa34","Type":"ContainerStarted","Data":"65a23352f000ee4059a52fdc0bf7866d9435ea14e8d820ff9c4e9a267390525e"} Nov 03 22:37:18.621303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:18.621249 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" podStartSLOduration=1.6212358180000002 podStartE2EDuration="1.621235818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:18.619374379 +0000 UTC m=+819.241682689" watchObservedRunningTime="2026-11-03 22:37:18.621235818 +0000 UTC m=+819.243544189" Nov 03 22:37:19.516699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:19.516659 5272 generic.go:356] "Generic (PLEG): container finished" podID="ed7755c6-a357-4cf8-b40c-70057cceaa34" containerID="e3637455cb7b94d1386f1112888e1dc334d3c815682473f6310ce6dabf7d2c17" exitCode=0 Nov 03 22:37:19.517136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:19.516744 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" event={"ID":"ed7755c6-a357-4cf8-b40c-70057cceaa34","Type":"ContainerDied","Data":"e3637455cb7b94d1386f1112888e1dc334d3c815682473f6310ce6dabf7d2c17"} Nov 03 22:37:20.662325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.662294 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:20.821357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.821319 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-xks-ca\") pod \"ed7755c6-a357-4cf8-b40c-70057cceaa34\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " Nov 03 22:37:20.821558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.821368 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7755c6-a357-4cf8-b40c-70057cceaa34-kube-api-access-wzbw9\" (UniqueName: \"kubernetes.io/projected/ed7755c6-a357-4cf8-b40c-70057cceaa34-kube-api-access-wzbw9\") pod \"ed7755c6-a357-4cf8-b40c-70057cceaa34\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " Nov 03 22:37:20.821558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.821431 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-openshift-ca\") pod \"ed7755c6-a357-4cf8-b40c-70057cceaa34\" (UID: \"ed7755c6-a357-4cf8-b40c-70057cceaa34\") " Nov 03 22:37:20.821843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.821797 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-openshift-ca" pod "ed7755c6-a357-4cf8-b40c-70057cceaa34" (UID: "ed7755c6-a357-4cf8-b40c-70057cceaa34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:20.823414 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.823394 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-xks-ca" pod "ed7755c6-a357-4cf8-b40c-70057cceaa34" (UID: "ed7755c6-a357-4cf8-b40c-70057cceaa34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:20.823512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.823494 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7755c6-a357-4cf8-b40c-70057cceaa34-kube-api-access-wzbw9" pod "ed7755c6-a357-4cf8-b40c-70057cceaa34" (UID: "ed7755c6-a357-4cf8-b40c-70057cceaa34"). InnerVolumeSpecName "kube-api-access-wzbw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:20.925740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.924827 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:20.925740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.924865 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7755c6-a357-4cf8-b40c-70057cceaa34-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:20.925740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:20.924880 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbw9\" (UniqueName: \"kubernetes.io/projected/ed7755c6-a357-4cf8-b40c-70057cceaa34-kube-api-access-wzbw9\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:21.529148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:21.529116 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" Nov 03 22:37:21.529326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:21.529113 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb" event={"ID":"ed7755c6-a357-4cf8-b40c-70057cceaa34","Type":"ContainerDied","Data":"65a23352f000ee4059a52fdc0bf7866d9435ea14e8d820ff9c4e9a267390525e"} Nov 03 22:37:21.529326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:21.529244 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65a23352f000ee4059a52fdc0bf7866d9435ea14e8d820ff9c4e9a267390525e" Nov 03 22:37:24.575827 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:37:24.575350 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:37:27.950649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:27.950613 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7"] Nov 03 22:37:27.951169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:27.951150 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ed7755c6-a357-4cf8-b40c-70057cceaa34" containerName="revoke-keys" Nov 03 22:37:27.951169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:27.951170 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7755c6-a357-4cf8-b40c-70057cceaa34" containerName="revoke-keys" Nov 03 22:37:27.951281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:27.951271 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7755c6-a357-4cf8-b40c-70057cceaa34" containerName="revoke-keys" Nov 03 22:37:27.954838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:27.954815 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.004376 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.004348 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:28.050946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.050919 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:28.053346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.053313 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:37:28.053346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.053340 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:28.053517 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.053348 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7"] Nov 03 22:37:28.064882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.064852 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.064977 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.064921 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq255\" (UniqueName: \"kubernetes.io/projected/12796649-4c89-4ae8-9f46-9a7f8a88c76e-kube-api-access-xq255\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.065045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.065000 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.166611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.166580 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.166779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.166635 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq255\" (UniqueName: \"kubernetes.io/projected/12796649-4c89-4ae8-9f46-9a7f8a88c76e-kube-api-access-xq255\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.166779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.166669 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.167378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.167351 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.169153 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.169133 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.224406 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.224337 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq255\" (UniqueName: \"kubernetes.io/projected/12796649-4c89-4ae8-9f46-9a7f8a88c76e-kube-api-access-xq255\") pod \"maas-api-revoke-sub-5fc898fd-v5xf7\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.267139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.267112 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:28.640817 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:28.640776 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7"] Nov 03 22:37:28.647749 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:28.647710 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12796649_4c89_4ae8_9f46_9a7f8a88c76e.slice/crio-b9c0d7e9cbaab60b2800b26ed1a7f6cb21aafaf17b161b3de1589101c6395361 WatchSource:0}: Error finding container b9c0d7e9cbaab60b2800b26ed1a7f6cb21aafaf17b161b3de1589101c6395361: Status 404 returned error can't find the container with id b9c0d7e9cbaab60b2800b26ed1a7f6cb21aafaf17b161b3de1589101c6395361 Nov 03 22:37:29.586341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:29.586300 5272 generic.go:356] "Generic (PLEG): container finished" podID="12796649-4c89-4ae8-9f46-9a7f8a88c76e" containerID="858c97fe94ad34cbb589db03361843b7c29dcc400aabdcb5b03a05b908dbb19b" exitCode=0 Nov 03 22:37:29.586790 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:29.586345 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" event={"ID":"12796649-4c89-4ae8-9f46-9a7f8a88c76e","Type":"ContainerDied","Data":"858c97fe94ad34cbb589db03361843b7c29dcc400aabdcb5b03a05b908dbb19b"} Nov 03 22:37:29.586790 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:29.586402 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" event={"ID":"12796649-4c89-4ae8-9f46-9a7f8a88c76e","Type":"ContainerStarted","Data":"b9c0d7e9cbaab60b2800b26ed1a7f6cb21aafaf17b161b3de1589101c6395361"} Nov 03 22:37:30.731214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.731188 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:30.896204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.896114 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-xks-ca\") pod \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " Nov 03 22:37:30.896204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.896199 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-openshift-ca\") pod \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " Nov 03 22:37:30.896448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.896336 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12796649-4c89-4ae8-9f46-9a7f8a88c76e-kube-api-access-xq255\" (UniqueName: \"kubernetes.io/projected/12796649-4c89-4ae8-9f46-9a7f8a88c76e-kube-api-access-xq255\") pod \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\" (UID: \"12796649-4c89-4ae8-9f46-9a7f8a88c76e\") " Nov 03 22:37:30.896596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.896574 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-openshift-ca" pod "12796649-4c89-4ae8-9f46-9a7f8a88c76e" (UID: "12796649-4c89-4ae8-9f46-9a7f8a88c76e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:30.896697 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.896675 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:30.898313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.898292 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12796649-4c89-4ae8-9f46-9a7f8a88c76e-kube-api-access-xq255" pod "12796649-4c89-4ae8-9f46-9a7f8a88c76e" (UID: "12796649-4c89-4ae8-9f46-9a7f8a88c76e"). InnerVolumeSpecName "kube-api-access-xq255". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:30.898382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.898364 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-xks-ca" pod "12796649-4c89-4ae8-9f46-9a7f8a88c76e" (UID: "12796649-4c89-4ae8-9f46-9a7f8a88c76e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:30.998130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.998093 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq255\" (UniqueName: \"kubernetes.io/projected/12796649-4c89-4ae8-9f46-9a7f8a88c76e-kube-api-access-xq255\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:30.998130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:30.998125 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12796649-4c89-4ae8-9f46-9a7f8a88c76e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:31.602012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:31.601986 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" Nov 03 22:37:31.602189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:31.602030 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7" event={"ID":"12796649-4c89-4ae8-9f46-9a7f8a88c76e","Type":"ContainerDied","Data":"b9c0d7e9cbaab60b2800b26ed1a7f6cb21aafaf17b161b3de1589101c6395361"} Nov 03 22:37:31.602189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:31.602083 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c0d7e9cbaab60b2800b26ed1a7f6cb21aafaf17b161b3de1589101c6395361" Nov 03 22:37:35.482949 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.482914 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5"] Nov 03 22:37:35.483432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.483418 5272 cpu_manager.go:414] "containerMap: removing container" podUID="12796649-4c89-4ae8-9f46-9a7f8a88c76e" containerName="revoke-keys" Nov 03 22:37:35.483477 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.483433 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12796649-4c89-4ae8-9f46-9a7f8a88c76e" containerName="revoke-keys" Nov 03 22:37:35.483511 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.483505 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="12796649-4c89-4ae8-9f46-9a7f8a88c76e" containerName="revoke-keys" Nov 03 22:37:35.487025 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.486994 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.584129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.584092 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:35.584129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.584110 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:35.584391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.584232 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:35.584391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.584379 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:37:35.586616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.586599 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5"] Nov 03 22:37:35.646222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.646195 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.646374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.646258 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.646374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.646356 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdc2l\" (UniqueName: \"kubernetes.io/projected/e7da17ed-305f-4c3f-8041-5f96776bde36-kube-api-access-hdc2l\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.747725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.747647 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.747725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.747719 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.747994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.747749 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc2l\" (UniqueName: \"kubernetes.io/projected/e7da17ed-305f-4c3f-8041-5f96776bde36-kube-api-access-hdc2l\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.748395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.748367 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.750133 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.750111 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:35.805476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:35.805445 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdc2l\" (UniqueName: \"kubernetes.io/projected/e7da17ed-305f-4c3f-8041-5f96776bde36-kube-api-access-hdc2l\") pod \"maas-api-revoke-sub-d3efff7f-86tf5\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:36.100527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:36.100495 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:36.470713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:36.470631 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5"] Nov 03 22:37:36.476631 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:36.476602 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7da17ed_305f_4c3f_8041_5f96776bde36.slice/crio-e95e98a9d1db951464b8eaa4eff3ab9ab135ab931f27b2deeff7370e7e318358 WatchSource:0}: Error finding container e95e98a9d1db951464b8eaa4eff3ab9ab135ab931f27b2deeff7370e7e318358: Status 404 returned error can't find the container with id e95e98a9d1db951464b8eaa4eff3ab9ab135ab931f27b2deeff7370e7e318358 Nov 03 22:37:36.632442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:36.632406 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" event={"ID":"e7da17ed-305f-4c3f-8041-5f96776bde36","Type":"ContainerStarted","Data":"e95e98a9d1db951464b8eaa4eff3ab9ab135ab931f27b2deeff7370e7e318358"} Nov 03 22:37:37.640839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.640733 5272 generic.go:356] "Generic (PLEG): container finished" podID="e7da17ed-305f-4c3f-8041-5f96776bde36" containerID="5310a7bd08d4988284c3c2be0d4f0a3765d9051bd8bcc4b6e0acbcd3d43c394d" exitCode=0 Nov 03 22:37:37.640839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.640769 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" event={"ID":"e7da17ed-305f-4c3f-8041-5f96776bde36","Type":"ContainerDied","Data":"5310a7bd08d4988284c3c2be0d4f0a3765d9051bd8bcc4b6e0acbcd3d43c394d"} Nov 03 22:37:37.644058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.644024 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk"] Nov 03 22:37:37.650868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.650833 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.739113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.739075 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk"] Nov 03 22:37:37.768429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.768391 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2wl\" (UniqueName: \"kubernetes.io/projected/f642e942-a25e-48cd-b026-7db11eafc9d0-kube-api-access-dt2wl\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.768597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.768511 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.768597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.768573 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.869641 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.869602 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2wl\" (UniqueName: \"kubernetes.io/projected/f642e942-a25e-48cd-b026-7db11eafc9d0-kube-api-access-dt2wl\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.869861 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.869687 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.869861 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.869733 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.870375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.870348 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.872073 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.872053 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.926768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.926702 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2wl\" (UniqueName: \"kubernetes.io/projected/f642e942-a25e-48cd-b026-7db11eafc9d0-kube-api-access-dt2wl\") pod \"maas-api-revoke-sub-0623e684-7cvhk\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:37.965272 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:37.965246 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:38.338088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.338051 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc"] Nov 03 22:37:38.343548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.343523 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk"] Nov 03 22:37:38.343668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.343610 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.349981 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:38.349953 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf642e942_a25e_48cd_b026_7db11eafc9d0.slice/crio-94c7a5f44acf3bcf294b3798eb85da9198345022b5b9f780e6a77157ff611ad4 WatchSource:0}: Error finding container 94c7a5f44acf3bcf294b3798eb85da9198345022b5b9f780e6a77157ff611ad4: Status 404 returned error can't find the container with id 94c7a5f44acf3bcf294b3798eb85da9198345022b5b9f780e6a77157ff611ad4 Nov 03 22:37:38.395995 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.395962 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc"] Nov 03 22:37:38.477325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.477287 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v8n9\" (UniqueName: \"kubernetes.io/projected/ed941de0-6465-4935-ac7a-6efa5bea567f-kube-api-access-4v8n9\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.477325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.477329 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.477547 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.477495 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.578677 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.578640 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.578873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.578705 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v8n9\" (UniqueName: \"kubernetes.io/projected/ed941de0-6465-4935-ac7a-6efa5bea567f-kube-api-access-4v8n9\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.578873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.578725 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.579364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.579341 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.581049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.581031 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.636070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.636000 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v8n9\" (UniqueName: \"kubernetes.io/projected/ed941de0-6465-4935-ac7a-6efa5bea567f-kube-api-access-4v8n9\") pod \"maas-api-revoke-sub-7fa9cf14-bbgkc\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.649245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.649209 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" event={"ID":"f642e942-a25e-48cd-b026-7db11eafc9d0","Type":"ContainerStarted","Data":"94c7a5f44acf3bcf294b3798eb85da9198345022b5b9f780e6a77157ff611ad4"} Nov 03 22:37:38.670575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.670537 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:38.781027 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.780998 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:38.882101 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.882068 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7da17ed-305f-4c3f-8041-5f96776bde36-kube-api-access-hdc2l\" (UniqueName: \"kubernetes.io/projected/e7da17ed-305f-4c3f-8041-5f96776bde36-kube-api-access-hdc2l\") pod \"e7da17ed-305f-4c3f-8041-5f96776bde36\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " Nov 03 22:37:38.882294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.882163 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-xks-ca\") pod \"e7da17ed-305f-4c3f-8041-5f96776bde36\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " Nov 03 22:37:38.882294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.882253 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-openshift-ca\") pod \"e7da17ed-305f-4c3f-8041-5f96776bde36\" (UID: \"e7da17ed-305f-4c3f-8041-5f96776bde36\") " Nov 03 22:37:38.882661 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.882631 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-openshift-ca" pod "e7da17ed-305f-4c3f-8041-5f96776bde36" (UID: "e7da17ed-305f-4c3f-8041-5f96776bde36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:38.884362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.884330 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-xks-ca" pod "e7da17ed-305f-4c3f-8041-5f96776bde36" (UID: "e7da17ed-305f-4c3f-8041-5f96776bde36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:38.884475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.884386 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7da17ed-305f-4c3f-8041-5f96776bde36-kube-api-access-hdc2l" pod "e7da17ed-305f-4c3f-8041-5f96776bde36" (UID: "e7da17ed-305f-4c3f-8041-5f96776bde36"). InnerVolumeSpecName "kube-api-access-hdc2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:38.983571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.983532 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdc2l\" (UniqueName: \"kubernetes.io/projected/e7da17ed-305f-4c3f-8041-5f96776bde36-kube-api-access-hdc2l\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:38.983571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.983566 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:38.983571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:38.983576 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7da17ed-305f-4c3f-8041-5f96776bde36-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:39.056505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.056467 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc"] Nov 03 22:37:39.062875 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:39.062846 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded941de0_6465_4935_ac7a_6efa5bea567f.slice/crio-e879a37d3920947d2f9fdd7a1b1716d8f64a1c510586f89f22e1287bfcce5ce4 WatchSource:0}: Error finding container e879a37d3920947d2f9fdd7a1b1716d8f64a1c510586f89f22e1287bfcce5ce4: Status 404 returned error can't find the container with id e879a37d3920947d2f9fdd7a1b1716d8f64a1c510586f89f22e1287bfcce5ce4 Nov 03 22:37:39.655145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.655114 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" Nov 03 22:37:39.655145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.655126 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5" event={"ID":"e7da17ed-305f-4c3f-8041-5f96776bde36","Type":"ContainerDied","Data":"e95e98a9d1db951464b8eaa4eff3ab9ab135ab931f27b2deeff7370e7e318358"} Nov 03 22:37:39.655576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.655174 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e95e98a9d1db951464b8eaa4eff3ab9ab135ab931f27b2deeff7370e7e318358" Nov 03 22:37:39.656630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.656603 5272 generic.go:356] "Generic (PLEG): container finished" podID="f642e942-a25e-48cd-b026-7db11eafc9d0" containerID="183ba58b7ffde9415223d38cbb7ca86bf3d74b1b0775ee3dfd8560def88b30db" exitCode=0 Nov 03 22:37:39.656756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.656697 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" event={"ID":"f642e942-a25e-48cd-b026-7db11eafc9d0","Type":"ContainerDied","Data":"183ba58b7ffde9415223d38cbb7ca86bf3d74b1b0775ee3dfd8560def88b30db"} Nov 03 22:37:39.658317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.658296 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" event={"ID":"ed941de0-6465-4935-ac7a-6efa5bea567f","Type":"ContainerStarted","Data":"8e41df334faf3a0a376debee1cc19e711416c711134ddf93e927be0099217737"} Nov 03 22:37:39.658404 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.658342 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" event={"ID":"ed941de0-6465-4935-ac7a-6efa5bea567f","Type":"ContainerStarted","Data":"e879a37d3920947d2f9fdd7a1b1716d8f64a1c510586f89f22e1287bfcce5ce4"} Nov 03 22:37:39.873502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:39.873403 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" podStartSLOduration=1.8733889160000001 podStartE2EDuration="1.873388916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:39.87229943 +0000 UTC m=+840.494607720" watchObservedRunningTime="2026-11-03 22:37:39.873388916 +0000 UTC m=+840.495697264" Nov 03 22:37:40.665987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.665950 5272 generic.go:356] "Generic (PLEG): container finished" podID="ed941de0-6465-4935-ac7a-6efa5bea567f" containerID="8e41df334faf3a0a376debee1cc19e711416c711134ddf93e927be0099217737" exitCode=0 Nov 03 22:37:40.666427 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.666042 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" event={"ID":"ed941de0-6465-4935-ac7a-6efa5bea567f","Type":"ContainerDied","Data":"8e41df334faf3a0a376debee1cc19e711416c711134ddf93e927be0099217737"} Nov 03 22:37:40.815886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.815860 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:40.904334 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.904295 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-xks-ca\") pod \"f642e942-a25e-48cd-b026-7db11eafc9d0\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " Nov 03 22:37:40.904521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.904503 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-openshift-ca\") pod \"f642e942-a25e-48cd-b026-7db11eafc9d0\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " Nov 03 22:37:40.904585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.904574 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f642e942-a25e-48cd-b026-7db11eafc9d0-kube-api-access-dt2wl\" (UniqueName: \"kubernetes.io/projected/f642e942-a25e-48cd-b026-7db11eafc9d0-kube-api-access-dt2wl\") pod \"f642e942-a25e-48cd-b026-7db11eafc9d0\" (UID: \"f642e942-a25e-48cd-b026-7db11eafc9d0\") " Nov 03 22:37:40.904974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.904944 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-openshift-ca" pod "f642e942-a25e-48cd-b026-7db11eafc9d0" (UID: "f642e942-a25e-48cd-b026-7db11eafc9d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:40.906445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.906423 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-xks-ca" pod "f642e942-a25e-48cd-b026-7db11eafc9d0" (UID: "f642e942-a25e-48cd-b026-7db11eafc9d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:40.906633 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:40.906618 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f642e942-a25e-48cd-b026-7db11eafc9d0-kube-api-access-dt2wl" pod "f642e942-a25e-48cd-b026-7db11eafc9d0" (UID: "f642e942-a25e-48cd-b026-7db11eafc9d0"). InnerVolumeSpecName "kube-api-access-dt2wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:41.005747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.005656 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:41.005747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.005688 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2wl\" (UniqueName: \"kubernetes.io/projected/f642e942-a25e-48cd-b026-7db11eafc9d0-kube-api-access-dt2wl\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:41.005747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.005700 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f642e942-a25e-48cd-b026-7db11eafc9d0-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:41.673028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.672995 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" Nov 03 22:37:41.673497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.673021 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk" event={"ID":"f642e942-a25e-48cd-b026-7db11eafc9d0","Type":"ContainerDied","Data":"94c7a5f44acf3bcf294b3798eb85da9198345022b5b9f780e6a77157ff611ad4"} Nov 03 22:37:41.673497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.673084 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94c7a5f44acf3bcf294b3798eb85da9198345022b5b9f780e6a77157ff611ad4" Nov 03 22:37:41.815626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.815596 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:41.915909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.915874 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-xks-ca\") pod \"ed941de0-6465-4935-ac7a-6efa5bea567f\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " Nov 03 22:37:41.916092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.915935 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-openshift-ca\") pod \"ed941de0-6465-4935-ac7a-6efa5bea567f\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " Nov 03 22:37:41.916092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.916080 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed941de0-6465-4935-ac7a-6efa5bea567f-kube-api-access-4v8n9\" (UniqueName: \"kubernetes.io/projected/ed941de0-6465-4935-ac7a-6efa5bea567f-kube-api-access-4v8n9\") pod \"ed941de0-6465-4935-ac7a-6efa5bea567f\" (UID: \"ed941de0-6465-4935-ac7a-6efa5bea567f\") " Nov 03 22:37:41.916352 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.916325 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-openshift-ca" pod "ed941de0-6465-4935-ac7a-6efa5bea567f" (UID: "ed941de0-6465-4935-ac7a-6efa5bea567f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:41.916444 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.916427 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:41.918125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.918100 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-xks-ca" pod "ed941de0-6465-4935-ac7a-6efa5bea567f" (UID: "ed941de0-6465-4935-ac7a-6efa5bea567f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:41.918125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:41.918117 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed941de0-6465-4935-ac7a-6efa5bea567f-kube-api-access-4v8n9" pod "ed941de0-6465-4935-ac7a-6efa5bea567f" (UID: "ed941de0-6465-4935-ac7a-6efa5bea567f"). InnerVolumeSpecName "kube-api-access-4v8n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:42.017717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:42.017675 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v8n9\" (UniqueName: \"kubernetes.io/projected/ed941de0-6465-4935-ac7a-6efa5bea567f-kube-api-access-4v8n9\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:42.017717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:42.017707 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed941de0-6465-4935-ac7a-6efa5bea567f-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:42.682638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:42.682601 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" Nov 03 22:37:42.682638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:42.682625 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc" event={"ID":"ed941de0-6465-4935-ac7a-6efa5bea567f","Type":"ContainerDied","Data":"e879a37d3920947d2f9fdd7a1b1716d8f64a1c510586f89f22e1287bfcce5ce4"} Nov 03 22:37:42.683227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:42.682677 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e879a37d3920947d2f9fdd7a1b1716d8f64a1c510586f89f22e1287bfcce5ce4" Nov 03 22:37:43.044320 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044279 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx"] Nov 03 22:37:43.044777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044764 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ed941de0-6465-4935-ac7a-6efa5bea567f" containerName="revoke-keys" Nov 03 22:37:43.044834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044777 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed941de0-6465-4935-ac7a-6efa5bea567f" containerName="revoke-keys" Nov 03 22:37:43.044834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044831 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e7da17ed-305f-4c3f-8041-5f96776bde36" containerName="revoke-keys" Nov 03 22:37:43.044925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044838 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7da17ed-305f-4c3f-8041-5f96776bde36" containerName="revoke-keys" Nov 03 22:37:43.044925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044844 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f642e942-a25e-48cd-b026-7db11eafc9d0" containerName="revoke-keys" Nov 03 22:37:43.044925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044849 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f642e942-a25e-48cd-b026-7db11eafc9d0" containerName="revoke-keys" Nov 03 22:37:43.044925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044919 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7da17ed-305f-4c3f-8041-5f96776bde36" containerName="revoke-keys" Nov 03 22:37:43.045062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044927 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f642e942-a25e-48cd-b026-7db11eafc9d0" containerName="revoke-keys" Nov 03 22:37:43.045062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.044936 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed941de0-6465-4935-ac7a-6efa5bea567f" containerName="revoke-keys" Nov 03 22:37:43.048574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.048548 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.100265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.100221 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:43.103064 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.103033 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx"] Nov 03 22:37:43.139984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.139952 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.139984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.139986 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8ns\" (UniqueName: \"kubernetes.io/projected/16d03a0a-a349-4700-b5a0-f0a286bd9469-kube-api-access-6t8ns\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.140200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.140097 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.145458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.145434 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:43.145561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.145434 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:43.145561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.145436 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:37:43.241175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.241145 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.241329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.241252 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.241329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.241272 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8ns\" (UniqueName: \"kubernetes.io/projected/16d03a0a-a349-4700-b5a0-f0a286bd9469-kube-api-access-6t8ns\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.241843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.241797 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.243662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.243637 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.298937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.298870 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8ns\" (UniqueName: \"kubernetes.io/projected/16d03a0a-a349-4700-b5a0-f0a286bd9469-kube-api-access-6t8ns\") pod \"maas-api-revoke-sub-b7e2ec86-9hbmx\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.362367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.362333 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:43.734257 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:43.734154 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx"] Nov 03 22:37:43.740700 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:43.740661 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d03a0a_a349_4700_b5a0_f0a286bd9469.slice/crio-730bb79955fc2e2b667cb41ac1f5c9ebbc9de6d0a0b96486dd388f3de8e528bb WatchSource:0}: Error finding container 730bb79955fc2e2b667cb41ac1f5c9ebbc9de6d0a0b96486dd388f3de8e528bb: Status 404 returned error can't find the container with id 730bb79955fc2e2b667cb41ac1f5c9ebbc9de6d0a0b96486dd388f3de8e528bb Nov 03 22:37:44.695770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:44.695740 5272 generic.go:356] "Generic (PLEG): container finished" podID="16d03a0a-a349-4700-b5a0-f0a286bd9469" containerID="30b2bf4c3b253612c4d46b9146478b9ea2e5cf569785042060e57c6b91d284e4" exitCode=0 Nov 03 22:37:44.695929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:44.695831 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" event={"ID":"16d03a0a-a349-4700-b5a0-f0a286bd9469","Type":"ContainerDied","Data":"30b2bf4c3b253612c4d46b9146478b9ea2e5cf569785042060e57c6b91d284e4"} Nov 03 22:37:44.695929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:44.695883 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" event={"ID":"16d03a0a-a349-4700-b5a0-f0a286bd9469","Type":"ContainerStarted","Data":"730bb79955fc2e2b667cb41ac1f5c9ebbc9de6d0a0b96486dd388f3de8e528bb"} Nov 03 22:37:45.541388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.541353 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r"] Nov 03 22:37:45.545599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.545578 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.561253 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.561225 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.561402 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.561277 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.561402 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.561304 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpvb\" (UniqueName: \"kubernetes.io/projected/6902bdbb-4f72-4029-821c-20e92c574765-kube-api-access-kcpvb\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.618386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.618357 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r"] Nov 03 22:37:45.661953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.661919 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.662119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.661970 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.662119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.661993 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpvb\" (UniqueName: \"kubernetes.io/projected/6902bdbb-4f72-4029-821c-20e92c574765-kube-api-access-kcpvb\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.662605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.662567 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.664402 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.664381 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.721895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.721859 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpvb\" (UniqueName: \"kubernetes.io/projected/6902bdbb-4f72-4029-821c-20e92c574765-kube-api-access-kcpvb\") pod \"maas-api-revoke-sub-0b4ee703-wnj2r\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.733983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.733954 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:37:45.843430 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.843407 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:45.858597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.858569 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:45.863626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.863597 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-xks-ca\") pod \"16d03a0a-a349-4700-b5a0-f0a286bd9469\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " Nov 03 22:37:45.863770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.863684 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d03a0a-a349-4700-b5a0-f0a286bd9469-kube-api-access-6t8ns\" (UniqueName: \"kubernetes.io/projected/16d03a0a-a349-4700-b5a0-f0a286bd9469-kube-api-access-6t8ns\") pod \"16d03a0a-a349-4700-b5a0-f0a286bd9469\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " Nov 03 22:37:45.865703 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.865678 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-xks-ca" pod "16d03a0a-a349-4700-b5a0-f0a286bd9469" (UID: "16d03a0a-a349-4700-b5a0-f0a286bd9469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:45.865909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.865886 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d03a0a-a349-4700-b5a0-f0a286bd9469-kube-api-access-6t8ns" pod "16d03a0a-a349-4700-b5a0-f0a286bd9469" (UID: "16d03a0a-a349-4700-b5a0-f0a286bd9469"). InnerVolumeSpecName "kube-api-access-6t8ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:45.964868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.964840 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-openshift-ca\") pod \"16d03a0a-a349-4700-b5a0-f0a286bd9469\" (UID: \"16d03a0a-a349-4700-b5a0-f0a286bd9469\") " Nov 03 22:37:45.965070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.965058 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:45.965131 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.965074 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t8ns\" (UniqueName: \"kubernetes.io/projected/16d03a0a-a349-4700-b5a0-f0a286bd9469-kube-api-access-6t8ns\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:45.965277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:45.965250 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-openshift-ca" pod "16d03a0a-a349-4700-b5a0-f0a286bd9469" (UID: "16d03a0a-a349-4700-b5a0-f0a286bd9469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:46.081397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.081307 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d03a0a-a349-4700-b5a0-f0a286bd9469-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:46.236563 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.236527 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r"] Nov 03 22:37:46.242783 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:46.242753 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6902bdbb_4f72_4029_821c_20e92c574765.slice/crio-369ba969fb87d5ecbdb14575344e0d4fe49845b073ee4166d7b6fd2456d3d3a7 WatchSource:0}: Error finding container 369ba969fb87d5ecbdb14575344e0d4fe49845b073ee4166d7b6fd2456d3d3a7: Status 404 returned error can't find the container with id 369ba969fb87d5ecbdb14575344e0d4fe49845b073ee4166d7b6fd2456d3d3a7 Nov 03 22:37:46.337878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.337416 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8"] Nov 03 22:37:46.339304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.338173 5272 cpu_manager.go:414] "containerMap: removing container" podUID="16d03a0a-a349-4700-b5a0-f0a286bd9469" containerName="revoke-keys" Nov 03 22:37:46.339304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.338200 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d03a0a-a349-4700-b5a0-f0a286bd9469" containerName="revoke-keys" Nov 03 22:37:46.339304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.338376 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d03a0a-a349-4700-b5a0-f0a286bd9469" containerName="revoke-keys" Nov 03 22:37:46.363057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.363024 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8"] Nov 03 22:37:46.363244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.363149 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.495750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.490910 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6tg\" (UniqueName: \"kubernetes.io/projected/4a423bd9-60fc-4dd1-84d4-f10d5c845621-kube-api-access-xv6tg\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.495750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.490990 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.495750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.491064 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.591948 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.591851 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.592433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.592012 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6tg\" (UniqueName: \"kubernetes.io/projected/4a423bd9-60fc-4dd1-84d4-f10d5c845621-kube-api-access-xv6tg\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.592433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.592132 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.592979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.592956 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.594284 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.594259 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.649902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.649870 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6tg\" (UniqueName: \"kubernetes.io/projected/4a423bd9-60fc-4dd1-84d4-f10d5c845621-kube-api-access-xv6tg\") pod \"maas-api-revoke-sub-09feca7a-pmlf8\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.684753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.684722 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:46.727827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.727203 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" event={"ID":"6902bdbb-4f72-4029-821c-20e92c574765","Type":"ContainerStarted","Data":"ba8dc500ea1222f97eaaeec1df08061dfa91448a91f7a466a6f3d77acec82d73"} Nov 03 22:37:46.727827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.727273 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" event={"ID":"6902bdbb-4f72-4029-821c-20e92c574765","Type":"ContainerStarted","Data":"369ba969fb87d5ecbdb14575344e0d4fe49845b073ee4166d7b6fd2456d3d3a7"} Nov 03 22:37:46.731108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.729918 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" event={"ID":"16d03a0a-a349-4700-b5a0-f0a286bd9469","Type":"ContainerDied","Data":"730bb79955fc2e2b667cb41ac1f5c9ebbc9de6d0a0b96486dd388f3de8e528bb"} Nov 03 22:37:46.731108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.729971 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="730bb79955fc2e2b667cb41ac1f5c9ebbc9de6d0a0b96486dd388f3de8e528bb" Nov 03 22:37:46.731108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.730035 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx" Nov 03 22:37:46.841192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:46.841132 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" podStartSLOduration=1.8411141469999999 podStartE2EDuration="1.841114147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:46.8406962 +0000 UTC m=+847.463004508" watchObservedRunningTime="2026-11-03 22:37:46.841114147 +0000 UTC m=+847.463422458" Nov 03 22:37:47.127349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:47.126758 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8"] Nov 03 22:37:47.738705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:47.738650 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" event={"ID":"4a423bd9-60fc-4dd1-84d4-f10d5c845621","Type":"ContainerStarted","Data":"c7cfe9151234802835599bc97c75b67eb3d2b8c0f801c369f05dcbec0ed67e90"} Nov 03 22:37:47.739147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:47.738721 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" event={"ID":"4a423bd9-60fc-4dd1-84d4-f10d5c845621","Type":"ContainerStarted","Data":"897054408e99ed3a5bc1eac0e9374a2e201ec3652bf2c935de8be9e3ff1afede"} Nov 03 22:37:47.740895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:47.740867 5272 generic.go:356] "Generic (PLEG): container finished" podID="6902bdbb-4f72-4029-821c-20e92c574765" containerID="ba8dc500ea1222f97eaaeec1df08061dfa91448a91f7a466a6f3d77acec82d73" exitCode=0 Nov 03 22:37:47.741040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:47.740980 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" event={"ID":"6902bdbb-4f72-4029-821c-20e92c574765","Type":"ContainerDied","Data":"ba8dc500ea1222f97eaaeec1df08061dfa91448a91f7a466a6f3d77acec82d73"} Nov 03 22:37:47.848449 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:47.848396 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" podStartSLOduration=1.848383585 podStartE2EDuration="1.848383585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:47.846774464 +0000 UTC m=+848.469082784" watchObservedRunningTime="2026-11-03 22:37:47.848383585 +0000 UTC m=+848.470691918" Nov 03 22:37:48.424267 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.424167 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8"] Nov 03 22:37:48.428147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.428124 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.432436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.432404 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.432559 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.432475 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.432621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.432594 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zbx\" (UniqueName: \"kubernetes.io/projected/bf234204-fcb1-4241-835b-f0a3d1e99b7b-kube-api-access-84zbx\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.480757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.480724 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8"] Nov 03 22:37:48.533887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.533852 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zbx\" (UniqueName: \"kubernetes.io/projected/bf234204-fcb1-4241-835b-f0a3d1e99b7b-kube-api-access-84zbx\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.534091 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.534048 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.534175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.534157 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.534860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.534829 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.536833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.536789 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.591564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.591526 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zbx\" (UniqueName: \"kubernetes.io/projected/bf234204-fcb1-4241-835b-f0a3d1e99b7b-kube-api-access-84zbx\") pod \"maas-api-revoke-sub-1c677a69-89hv8\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.741305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.741209 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:48.748615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.748582 5272 generic.go:356] "Generic (PLEG): container finished" podID="4a423bd9-60fc-4dd1-84d4-f10d5c845621" containerID="c7cfe9151234802835599bc97c75b67eb3d2b8c0f801c369f05dcbec0ed67e90" exitCode=0 Nov 03 22:37:48.748736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.748663 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" event={"ID":"4a423bd9-60fc-4dd1-84d4-f10d5c845621","Type":"ContainerDied","Data":"c7cfe9151234802835599bc97c75b67eb3d2b8c0f801c369f05dcbec0ed67e90"} Nov 03 22:37:48.757006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.756960 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9"] Nov 03 22:37:48.761414 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.761387 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.824041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.823996 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9"] Nov 03 22:37:48.836365 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.836336 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.836458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.836377 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.836458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.836445 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqh8g\" (UniqueName: \"kubernetes.io/projected/e47dae07-22e1-4ce8-a6ab-19750b713bb1-kube-api-access-bqh8g\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.934063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.934034 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:48.937258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.937238 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqh8g\" (UniqueName: \"kubernetes.io/projected/e47dae07-22e1-4ce8-a6ab-19750b713bb1-kube-api-access-bqh8g\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.937362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.937346 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.937398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.937383 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.938106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.938088 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.939727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.939702 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.994478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.994403 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n"] Nov 03 22:37:48.994942 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.994926 5272 cpu_manager.go:414] "containerMap: removing container" podUID="6902bdbb-4f72-4029-821c-20e92c574765" containerName="revoke-keys" Nov 03 22:37:48.994997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.994942 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6902bdbb-4f72-4029-821c-20e92c574765" containerName="revoke-keys" Nov 03 22:37:48.995031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.995026 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="6902bdbb-4f72-4029-821c-20e92c574765" containerName="revoke-keys" Nov 03 22:37:48.995511 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.995492 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqh8g\" (UniqueName: \"kubernetes.io/projected/e47dae07-22e1-4ce8-a6ab-19750b713bb1-kube-api-access-bqh8g\") pod \"maas-api-revoke-sub-ad95826b-xkjv9\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:48.998576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:48.998557 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.015703 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.015676 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl"] Nov 03 22:37:49.019716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.019690 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.038263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038231 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-openshift-ca\") pod \"6902bdbb-4f72-4029-821c-20e92c574765\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " Nov 03 22:37:49.038448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038428 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6902bdbb-4f72-4029-821c-20e92c574765-kube-api-access-kcpvb\" (UniqueName: \"kubernetes.io/projected/6902bdbb-4f72-4029-821c-20e92c574765-kube-api-access-kcpvb\") pod \"6902bdbb-4f72-4029-821c-20e92c574765\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " Nov 03 22:37:49.038885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038528 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-xks-ca\") pod \"6902bdbb-4f72-4029-821c-20e92c574765\" (UID: \"6902bdbb-4f72-4029-821c-20e92c574765\") " Nov 03 22:37:49.038885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038583 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-openshift-ca" pod "6902bdbb-4f72-4029-821c-20e92c574765" (UID: "6902bdbb-4f72-4029-821c-20e92c574765"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:49.038885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038734 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.038885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038784 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038897 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038919 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.038939 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039009 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039092 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039138 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064709c-6647-4160-98ab-3801cdd1a76e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039170 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039223 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039273 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039339 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039368 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064709c-6647-4160-98ab-3801cdd1a76e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039400 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039520 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039571 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncxp\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-kube-api-access-9ncxp\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.039786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039604 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzkp\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-kube-api-access-dtzkp\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039636 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.039786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.039689 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:49.040722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.040700 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6902bdbb-4f72-4029-821c-20e92c574765-kube-api-access-kcpvb" pod "6902bdbb-4f72-4029-821c-20e92c574765" (UID: "6902bdbb-4f72-4029-821c-20e92c574765"). InnerVolumeSpecName "kube-api-access-kcpvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:49.041011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.040988 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-xks-ca" pod "6902bdbb-4f72-4029-821c-20e92c574765" (UID: "6902bdbb-4f72-4029-821c-20e92c574765"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:49.057355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.057326 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n"] Nov 03 22:37:49.082887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.082753 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl"] Nov 03 22:37:49.087280 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.087234 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:49.124947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.124904 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8"] Nov 03 22:37:49.129585 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:49.129556 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf234204_fcb1_4241_835b_f0a3d1e99b7b.slice/crio-94ac77f806f22a666fbe4ba209cfeb84735bb5671dc17540c1353264b1b3cfa2 WatchSource:0}: Error finding container 94ac77f806f22a666fbe4ba209cfeb84735bb5671dc17540c1353264b1b3cfa2: Status 404 returned error can't find the container with id 94ac77f806f22a666fbe4ba209cfeb84735bb5671dc17540c1353264b1b3cfa2 Nov 03 22:37:49.146504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146468 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064709c-6647-4160-98ab-3801cdd1a76e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.146776 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146746 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.146903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146793 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.146903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146839 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.146903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146869 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.146903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146892 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064709c-6647-4160-98ab-3801cdd1a76e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.147132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146934 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.147132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.146969 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147005 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncxp\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-kube-api-access-9ncxp\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147032 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzkp\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-kube-api-access-dtzkp\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.147132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147059 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.147132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147113 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147159 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.147417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147164 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.147417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147199 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147236 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147262 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147287 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147355 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.147753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147418 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.147753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147444 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpvb\" (UniqueName: \"kubernetes.io/projected/6902bdbb-4f72-4029-821c-20e92c574765-kube-api-access-kcpvb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:49.147753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147465 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6902bdbb-4f72-4029-821c-20e92c574765-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:49.147937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.147914 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.148216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.148197 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.148465 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.148445 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.148573 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.148550 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064709c-6647-4160-98ab-3801cdd1a76e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.149093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.149066 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.149189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.149167 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.149345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.149323 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.149698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.149671 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.150465 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.150443 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064709c-6647-4160-98ab-3801cdd1a76e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.151179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.151157 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.151562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.151542 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.153156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.153111 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.206112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.206064 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.206317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.206125 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzkp\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-kube-api-access-dtzkp\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mm95n\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.206317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.206168 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncxp\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-kube-api-access-9ncxp\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.206455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.206409 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dvtvl\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.314193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.314159 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:49.333197 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.333167 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:49.480413 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.480370 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9"] Nov 03 22:37:49.485615 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:49.485580 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47dae07_22e1_4ce8_a6ab_19750b713bb1.slice/crio-8e5b93a245d99d1c4c835622d050660069fdf5853be4767dd205816f9e952626 WatchSource:0}: Error finding container 8e5b93a245d99d1c4c835622d050660069fdf5853be4767dd205816f9e952626: Status 404 returned error can't find the container with id 8e5b93a245d99d1c4c835622d050660069fdf5853be4767dd205816f9e952626 Nov 03 22:37:49.742646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.742614 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl"] Nov 03 22:37:49.748634 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:49.748606 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95a92be_6793_4ea9_be82_fdbeb4f54ca5.slice/crio-c015d3f35fcac43bfe13e5f3aa9572d317d8dfc97a5be5d199ee13f51d36ba73 WatchSource:0}: Error finding container c015d3f35fcac43bfe13e5f3aa9572d317d8dfc97a5be5d199ee13f51d36ba73: Status 404 returned error can't find the container with id c015d3f35fcac43bfe13e5f3aa9572d317d8dfc97a5be5d199ee13f51d36ba73 Nov 03 22:37:49.751402 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.751360 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.751477 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.751453 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.751512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.751496 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.759090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.759062 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" event={"ID":"bf234204-fcb1-4241-835b-f0a3d1e99b7b","Type":"ContainerStarted","Data":"8b2203976084c504c3b0ac16fb844b27797abeadd204ff4c38a850f7d1a5bf0b"} Nov 03 22:37:49.759208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.759127 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" event={"ID":"bf234204-fcb1-4241-835b-f0a3d1e99b7b","Type":"ContainerStarted","Data":"94ac77f806f22a666fbe4ba209cfeb84735bb5671dc17540c1353264b1b3cfa2"} Nov 03 22:37:49.761082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.761060 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" Nov 03 22:37:49.761177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.761062 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r" event={"ID":"6902bdbb-4f72-4029-821c-20e92c574765","Type":"ContainerDied","Data":"369ba969fb87d5ecbdb14575344e0d4fe49845b073ee4166d7b6fd2456d3d3a7"} Nov 03 22:37:49.761232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.761188 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369ba969fb87d5ecbdb14575344e0d4fe49845b073ee4166d7b6fd2456d3d3a7" Nov 03 22:37:49.762850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.762828 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" event={"ID":"e47dae07-22e1-4ce8-a6ab-19750b713bb1","Type":"ContainerStarted","Data":"8e5b93a245d99d1c4c835622d050660069fdf5853be4767dd205816f9e952626"} Nov 03 22:37:49.764223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.764201 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" event={"ID":"e95a92be-6793-4ea9-be82-fdbeb4f54ca5","Type":"ContainerStarted","Data":"c015d3f35fcac43bfe13e5f3aa9572d317d8dfc97a5be5d199ee13f51d36ba73"} Nov 03 22:37:49.794101 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.794062 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n"] Nov 03 22:37:49.809534 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:49.809501 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7064709c_6647_4160_98ab_3801cdd1a76e.slice/crio-eff1aa8261682231d0142e638e7298348bdf82732901a68036ebd43e15be0415 WatchSource:0}: Error finding container eff1aa8261682231d0142e638e7298348bdf82732901a68036ebd43e15be0415: Status 404 returned error can't find the container with id eff1aa8261682231d0142e638e7298348bdf82732901a68036ebd43e15be0415 Nov 03 22:37:49.812412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.811904 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.812412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.811990 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.812412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.812026 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:49.871790 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.871720 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" podStartSLOduration=1.871700879 podStartE2EDuration="1.871700879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:49.87007236 +0000 UTC m=+850.492380669" watchObservedRunningTime="2026-11-03 22:37:49.871700879 +0000 UTC m=+850.494009187" Nov 03 22:37:49.927985 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.927956 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:49.956528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.956493 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-openshift-ca\") pod \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " Nov 03 22:37:49.956791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.956771 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a423bd9-60fc-4dd1-84d4-f10d5c845621-kube-api-access-xv6tg\" (UniqueName: \"kubernetes.io/projected/4a423bd9-60fc-4dd1-84d4-f10d5c845621-kube-api-access-xv6tg\") pod \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " Nov 03 22:37:49.956995 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.956978 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-xks-ca\") pod \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\" (UID: \"4a423bd9-60fc-4dd1-84d4-f10d5c845621\") " Nov 03 22:37:49.958085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.958049 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-openshift-ca" pod "4a423bd9-60fc-4dd1-84d4-f10d5c845621" (UID: "4a423bd9-60fc-4dd1-84d4-f10d5c845621"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:49.959926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.959902 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a423bd9-60fc-4dd1-84d4-f10d5c845621-kube-api-access-xv6tg" pod "4a423bd9-60fc-4dd1-84d4-f10d5c845621" (UID: "4a423bd9-60fc-4dd1-84d4-f10d5c845621"). InnerVolumeSpecName "kube-api-access-xv6tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:49.960128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:49.960082 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-xks-ca" pod "4a423bd9-60fc-4dd1-84d4-f10d5c845621" (UID: "4a423bd9-60fc-4dd1-84d4-f10d5c845621"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:50.058797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.058759 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6tg\" (UniqueName: \"kubernetes.io/projected/4a423bd9-60fc-4dd1-84d4-f10d5c845621-kube-api-access-xv6tg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:50.058962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.058840 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:50.058962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.058858 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a423bd9-60fc-4dd1-84d4-f10d5c845621-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:50.784634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.784562 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" event={"ID":"e95a92be-6793-4ea9-be82-fdbeb4f54ca5","Type":"ContainerStarted","Data":"dcfb529d3145116cf3cd5f620d55083430c51f0c784a09767ff149f5daae5cd7"} Nov 03 22:37:50.789605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.789571 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" event={"ID":"4a423bd9-60fc-4dd1-84d4-f10d5c845621","Type":"ContainerDied","Data":"897054408e99ed3a5bc1eac0e9374a2e201ec3652bf2c935de8be9e3ff1afede"} Nov 03 22:37:50.789913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.789792 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="897054408e99ed3a5bc1eac0e9374a2e201ec3652bf2c935de8be9e3ff1afede" Nov 03 22:37:50.790037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.787790 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8" Nov 03 22:37:50.797999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.797669 5272 generic.go:356] "Generic (PLEG): container finished" podID="bf234204-fcb1-4241-835b-f0a3d1e99b7b" containerID="8b2203976084c504c3b0ac16fb844b27797abeadd204ff4c38a850f7d1a5bf0b" exitCode=0 Nov 03 22:37:50.797999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.797794 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" event={"ID":"bf234204-fcb1-4241-835b-f0a3d1e99b7b","Type":"ContainerDied","Data":"8b2203976084c504c3b0ac16fb844b27797abeadd204ff4c38a850f7d1a5bf0b"} Nov 03 22:37:50.800910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.800559 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" event={"ID":"7064709c-6647-4160-98ab-3801cdd1a76e","Type":"ContainerStarted","Data":"6b3bcb90747eef4c07f9698286148121159fe0d0b9272372e22fa403bd1de2ac"} Nov 03 22:37:50.800910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.800626 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" event={"ID":"7064709c-6647-4160-98ab-3801cdd1a76e","Type":"ContainerStarted","Data":"eff1aa8261682231d0142e638e7298348bdf82732901a68036ebd43e15be0415"} Nov 03 22:37:50.803867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.803836 5272 generic.go:356] "Generic (PLEG): container finished" podID="e47dae07-22e1-4ce8-a6ab-19750b713bb1" containerID="bbad2d60c25c730311d1e1f70e9d26693048634d2c224337c2668790f5127178" exitCode=0 Nov 03 22:37:50.804017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.803976 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" event={"ID":"e47dae07-22e1-4ce8-a6ab-19750b713bb1","Type":"ContainerDied","Data":"bbad2d60c25c730311d1e1f70e9d26693048634d2c224337c2668790f5127178"} Nov 03 22:37:50.935729 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:50.935674 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podStartSLOduration=2.935659022 podStartE2EDuration="2.935659022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:50.900221064 +0000 UTC m=+851.522529373" watchObservedRunningTime="2026-11-03 22:37:50.935659022 +0000 UTC m=+851.557967330" Nov 03 22:37:51.016681 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:51.016556 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podStartSLOduration=3.016540796 podStartE2EDuration="3.016540796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:51.015057432 +0000 UTC m=+851.637365741" watchObservedRunningTime="2026-11-03 22:37:51.016540796 +0000 UTC m=+851.638849100" Nov 03 22:37:51.314420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:51.314384 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:51.316322 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:51.316283 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:37:51.316499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:51.316472 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:37:51.333560 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:51.333530 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:51.334779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:51.334751 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:37:51.334867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:51.334837 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:37:52.112878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.112854 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:52.114908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.114889 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:52.207553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.207516 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-openshift-ca\") pod \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " Nov 03 22:37:52.207732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.207605 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf234204-fcb1-4241-835b-f0a3d1e99b7b-kube-api-access-84zbx\" (UniqueName: \"kubernetes.io/projected/bf234204-fcb1-4241-835b-f0a3d1e99b7b-kube-api-access-84zbx\") pod \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " Nov 03 22:37:52.207732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.207670 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-openshift-ca\") pod \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " Nov 03 22:37:52.207732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.207709 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47dae07-22e1-4ce8-a6ab-19750b713bb1-kube-api-access-bqh8g\" (UniqueName: \"kubernetes.io/projected/e47dae07-22e1-4ce8-a6ab-19750b713bb1-kube-api-access-bqh8g\") pod \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " Nov 03 22:37:52.207938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.207773 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-xks-ca\") pod \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\" (UID: \"e47dae07-22e1-4ce8-a6ab-19750b713bb1\") " Nov 03 22:37:52.207938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.207847 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-xks-ca\") pod \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\" (UID: \"bf234204-fcb1-4241-835b-f0a3d1e99b7b\") " Nov 03 22:37:52.208623 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.208595 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-openshift-ca" pod "e47dae07-22e1-4ce8-a6ab-19750b713bb1" (UID: "e47dae07-22e1-4ce8-a6ab-19750b713bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:52.208752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.208630 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-openshift-ca" pod "bf234204-fcb1-4241-835b-f0a3d1e99b7b" (UID: "bf234204-fcb1-4241-835b-f0a3d1e99b7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:52.210854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.210781 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-xks-ca" pod "bf234204-fcb1-4241-835b-f0a3d1e99b7b" (UID: "bf234204-fcb1-4241-835b-f0a3d1e99b7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:52.211190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.211164 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf234204-fcb1-4241-835b-f0a3d1e99b7b-kube-api-access-84zbx" pod "bf234204-fcb1-4241-835b-f0a3d1e99b7b" (UID: "bf234204-fcb1-4241-835b-f0a3d1e99b7b"). InnerVolumeSpecName "kube-api-access-84zbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:52.211844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.211796 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-xks-ca" pod "e47dae07-22e1-4ce8-a6ab-19750b713bb1" (UID: "e47dae07-22e1-4ce8-a6ab-19750b713bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:52.212052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.212029 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47dae07-22e1-4ce8-a6ab-19750b713bb1-kube-api-access-bqh8g" pod "e47dae07-22e1-4ce8-a6ab-19750b713bb1" (UID: "e47dae07-22e1-4ce8-a6ab-19750b713bb1"). InnerVolumeSpecName "kube-api-access-bqh8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:52.309712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.309675 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqh8g\" (UniqueName: \"kubernetes.io/projected/e47dae07-22e1-4ce8-a6ab-19750b713bb1-kube-api-access-bqh8g\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:52.309712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.309710 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:52.309943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.309724 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:52.309943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.309733 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf234204-fcb1-4241-835b-f0a3d1e99b7b-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:52.309943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.309744 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84zbx\" (UniqueName: \"kubernetes.io/projected/bf234204-fcb1-4241-835b-f0a3d1e99b7b-kube-api-access-84zbx\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:52.309943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.309753 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47dae07-22e1-4ce8-a6ab-19750b713bb1-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:52.314609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.314580 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:37:52.314667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.314644 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:37:52.334442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.334374 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:37:52.334442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.334422 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:37:52.817475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.817435 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" Nov 03 22:37:52.817475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.817461 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8" event={"ID":"bf234204-fcb1-4241-835b-f0a3d1e99b7b","Type":"ContainerDied","Data":"94ac77f806f22a666fbe4ba209cfeb84735bb5671dc17540c1353264b1b3cfa2"} Nov 03 22:37:52.817672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.817515 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94ac77f806f22a666fbe4ba209cfeb84735bb5671dc17540c1353264b1b3cfa2" Nov 03 22:37:52.820067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.820040 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" Nov 03 22:37:52.820974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.820951 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9" event={"ID":"e47dae07-22e1-4ce8-a6ab-19750b713bb1","Type":"ContainerDied","Data":"8e5b93a245d99d1c4c835622d050660069fdf5853be4767dd205816f9e952626"} Nov 03 22:37:52.821069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:52.820992 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e5b93a245d99d1c4c835622d050660069fdf5853be4767dd205816f9e952626" Nov 03 22:37:53.314596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.314544 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:37:53.315037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.314609 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:37:53.333952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.333922 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:37:53.334073 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.333972 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:37:53.660888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.643122 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh"] Nov 03 22:37:53.661946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.661912 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4a423bd9-60fc-4dd1-84d4-f10d5c845621" containerName="revoke-keys" Nov 03 22:37:53.662128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662113 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a423bd9-60fc-4dd1-84d4-f10d5c845621" containerName="revoke-keys" Nov 03 22:37:53.662212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662201 5272 cpu_manager.go:414] "containerMap: removing container" podUID="bf234204-fcb1-4241-835b-f0a3d1e99b7b" containerName="revoke-keys" Nov 03 22:37:53.662285 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662274 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf234204-fcb1-4241-835b-f0a3d1e99b7b" containerName="revoke-keys" Nov 03 22:37:53.662365 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662354 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e47dae07-22e1-4ce8-a6ab-19750b713bb1" containerName="revoke-keys" Nov 03 22:37:53.662442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662431 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47dae07-22e1-4ce8-a6ab-19750b713bb1" containerName="revoke-keys" Nov 03 22:37:53.662696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662683 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a423bd9-60fc-4dd1-84d4-f10d5c845621" containerName="revoke-keys" Nov 03 22:37:53.662783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662773 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf234204-fcb1-4241-835b-f0a3d1e99b7b" containerName="revoke-keys" Nov 03 22:37:53.662897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.662886 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47dae07-22e1-4ce8-a6ab-19750b713bb1" containerName="revoke-keys" Nov 03 22:37:53.682090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.682059 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.708754 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.708732 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh"] Nov 03 22:37:53.727077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.727045 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-home\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.727264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.727140 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tmp-dir\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.727264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.727175 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-model-cache\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.727264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.727246 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmnnk\" (UniqueName: \"kubernetes.io/projected/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-kube-api-access-gmnnk\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.727358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.727316 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tls-certs\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.727388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.727367 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-dshm\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.786830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.784178 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-68fbd1\"/\"kube-root-ca.crt\"" Nov 03 22:37:53.786830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.784297 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-68fbd1\"/\"openshift-service-ca.crt\"" Nov 03 22:37:53.786830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.785153 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-68fbd1\"/\"default-dockercfg-5blrx\"" Nov 03 22:37:53.786830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.785488 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-68fbd1\"/\"distinct-w0-68fbd1-kserve-self-signed-certs\"" Nov 03 22:37:53.829824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.828409 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tmp-dir\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.829824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.828478 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-model-cache\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.829824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.828565 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmnnk\" (UniqueName: \"kubernetes.io/projected/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-kube-api-access-gmnnk\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.829824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.828649 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tls-certs\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.829824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.828719 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-dshm\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.834017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.832859 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-home\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.834017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.833776 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tmp-dir\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.834256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.834094 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-model-cache\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.834415 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.834384 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-home\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.836478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.836458 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-dshm\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.837648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.837624 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tls-certs\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:53.903229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:53.903185 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmnnk\" (UniqueName: \"kubernetes.io/projected/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-kube-api-access-gmnnk\") pod \"distinct-w0-68fbd1-kserve-77855d978d-4zsbh\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:54.009420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.009328 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:54.314530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.314489 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:37:54.314685 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.314547 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:37:54.334016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.333988 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:37:54.334129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.334054 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:37:54.497878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.486585 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh"] Nov 03 22:37:54.500290 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:54.500247 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6feb3c85_e3ef_450c_bf49_6642e0e4ddb0.slice/crio-ef5fae090b9acdee2249665f64f40716eff1b4ff8027c1a40306ef3a8d14f058 WatchSource:0}: Error finding container ef5fae090b9acdee2249665f64f40716eff1b4ff8027c1a40306ef3a8d14f058: Status 404 returned error can't find the container with id ef5fae090b9acdee2249665f64f40716eff1b4ff8027c1a40306ef3a8d14f058 Nov 03 22:37:54.521892 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:37:54.521861 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:37:54.875890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.867284 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" event={"ID":"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0","Type":"ContainerStarted","Data":"b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d"} Nov 03 22:37:54.875890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.867340 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" event={"ID":"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0","Type":"ContainerStarted","Data":"ef5fae090b9acdee2249665f64f40716eff1b4ff8027c1a40306ef3a8d14f058"} Nov 03 22:37:54.998609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:54.998525 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podStartSLOduration=1.998501343 podStartE2EDuration="1.998501343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:54.983086382 +0000 UTC m=+855.605394691" watchObservedRunningTime="2026-11-03 22:37:54.998501343 +0000 UTC m=+855.620809652" Nov 03 22:37:55.010542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:55.010511 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:37:55.315277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:55.315240 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:37:55.315682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:55.315302 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:37:55.334325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:55.334299 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:37:55.334412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:55.334344 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:37:56.315372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:56.315323 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:37:56.316146 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:56.316095 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:37:56.334354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:56.334328 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:37:56.334473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:56.334370 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:37:57.315123 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:57.315081 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:37:57.315308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:57.315157 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:37:57.334827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:57.334595 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:37:57.334827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:57.334661 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:37:58.807771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:58.807728 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:58.808567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:58.808523 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:58.810167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:58.810146 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:58.810622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:58.810601 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:37:58.811263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:58.811043 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:58.812324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:58.812302 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:37:59.094344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.094259 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk"] Nov 03 22:37:59.098611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.098589 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.150646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.150619 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:59.151846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.151824 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:37:59.152143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.152120 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:59.152275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.152187 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:59.172542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.172504 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk"] Nov 03 22:37:59.253150 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.253119 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np9bw\" (UniqueName: \"kubernetes.io/projected/5a61f3ee-8181-4669-a191-0d6dba217449-kube-api-access-np9bw\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.253366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.253176 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.253366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.253314 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.355055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.354960 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np9bw\" (UniqueName: \"kubernetes.io/projected/5a61f3ee-8181-4669-a191-0d6dba217449-kube-api-access-np9bw\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.355055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.355029 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.355295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.355093 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.355910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.355876 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.357367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.357348 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.412575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.412545 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np9bw\" (UniqueName: \"kubernetes.io/projected/5a61f3ee-8181-4669-a191-0d6dba217449-kube-api-access-np9bw\") pod \"maas-api-revoke-sub-5cb384dd-l89pk\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.412713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.412700 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:37:59.787680 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.787648 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk"] Nov 03 22:37:59.793718 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:37:59.793692 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a61f3ee_8181_4669_a191_0d6dba217449.slice/crio-dd5d7d58ddc1ab0f87dd1b37831933802d84a7d4fc74a2011e121ae84718455c WatchSource:0}: Error finding container dd5d7d58ddc1ab0f87dd1b37831933802d84a7d4fc74a2011e121ae84718455c: Status 404 returned error can't find the container with id dd5d7d58ddc1ab0f87dd1b37831933802d84a7d4fc74a2011e121ae84718455c Nov 03 22:37:59.927040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:37:59.926991 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" event={"ID":"5a61f3ee-8181-4669-a191-0d6dba217449","Type":"ContainerStarted","Data":"dd5d7d58ddc1ab0f87dd1b37831933802d84a7d4fc74a2011e121ae84718455c"} Nov 03 22:38:00.935939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:00.935901 5272 generic.go:356] "Generic (PLEG): container finished" podID="5a61f3ee-8181-4669-a191-0d6dba217449" containerID="2268bcf89acd5e0d4ac3fd4b9f5bc984f641f06dd64487333b73bce975e82cba" exitCode=0 Nov 03 22:38:00.936364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:00.935948 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" event={"ID":"5a61f3ee-8181-4669-a191-0d6dba217449","Type":"ContainerDied","Data":"2268bcf89acd5e0d4ac3fd4b9f5bc984f641f06dd64487333b73bce975e82cba"} Nov 03 22:38:02.104193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.104144 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:38:02.187159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.187126 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-openshift-ca\") pod \"5a61f3ee-8181-4669-a191-0d6dba217449\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " Nov 03 22:38:02.187367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.187289 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a61f3ee-8181-4669-a191-0d6dba217449-kube-api-access-np9bw\" (UniqueName: \"kubernetes.io/projected/5a61f3ee-8181-4669-a191-0d6dba217449-kube-api-access-np9bw\") pod \"5a61f3ee-8181-4669-a191-0d6dba217449\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " Nov 03 22:38:02.187367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.187338 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-xks-ca\") pod \"5a61f3ee-8181-4669-a191-0d6dba217449\" (UID: \"5a61f3ee-8181-4669-a191-0d6dba217449\") " Nov 03 22:38:02.187533 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.187506 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-openshift-ca" pod "5a61f3ee-8181-4669-a191-0d6dba217449" (UID: "5a61f3ee-8181-4669-a191-0d6dba217449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:02.187838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.187815 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:02.189571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.189540 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a61f3ee-8181-4669-a191-0d6dba217449-kube-api-access-np9bw" pod "5a61f3ee-8181-4669-a191-0d6dba217449" (UID: "5a61f3ee-8181-4669-a191-0d6dba217449"). InnerVolumeSpecName "kube-api-access-np9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:02.189868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.189847 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-xks-ca" pod "5a61f3ee-8181-4669-a191-0d6dba217449" (UID: "5a61f3ee-8181-4669-a191-0d6dba217449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:02.289287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.289248 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np9bw\" (UniqueName: \"kubernetes.io/projected/5a61f3ee-8181-4669-a191-0d6dba217449-kube-api-access-np9bw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:02.289287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.289280 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a61f3ee-8181-4669-a191-0d6dba217449-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:02.473654 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.473562 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk"] Nov 03 22:38:02.474169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.474152 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5a61f3ee-8181-4669-a191-0d6dba217449" containerName="revoke-keys" Nov 03 22:38:02.474258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.474169 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a61f3ee-8181-4669-a191-0d6dba217449" containerName="revoke-keys" Nov 03 22:38:02.474258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.474252 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a61f3ee-8181-4669-a191-0d6dba217449" containerName="revoke-keys" Nov 03 22:38:02.477837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.477815 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.532792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.532714 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk"] Nov 03 22:38:02.592490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.592452 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hbp\" (UniqueName: \"kubernetes.io/projected/b8f6af3c-e6b8-412e-afad-296da72a59c7-kube-api-access-78hbp\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.592713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.592524 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.592713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.592646 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.694169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.694132 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.694340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.694259 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hbp\" (UniqueName: \"kubernetes.io/projected/b8f6af3c-e6b8-412e-afad-296da72a59c7-kube-api-access-78hbp\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.694340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.694289 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.696840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.695104 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.705160 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.703258 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.752081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.752010 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hbp\" (UniqueName: \"kubernetes.io/projected/b8f6af3c-e6b8-412e-afad-296da72a59c7-kube-api-access-78hbp\") pod \"maas-api-revoke-sub-67ef9fac-lsdmk\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.792007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.791979 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:02.951734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.951705 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" Nov 03 22:38:02.951929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.951744 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk" event={"ID":"5a61f3ee-8181-4669-a191-0d6dba217449","Type":"ContainerDied","Data":"dd5d7d58ddc1ab0f87dd1b37831933802d84a7d4fc74a2011e121ae84718455c"} Nov 03 22:38:02.951929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:02.951793 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd5d7d58ddc1ab0f87dd1b37831933802d84a7d4fc74a2011e121ae84718455c" Nov 03 22:38:03.164982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:03.164945 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk"] Nov 03 22:38:03.170969 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:38:03.170939 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f6af3c_e6b8_412e_afad_296da72a59c7.slice/crio-d7cddc583947a936caa034d25d3d79dcbb0b4713e7f51fdbc33ed5746c83a9f6 WatchSource:0}: Error finding container d7cddc583947a936caa034d25d3d79dcbb0b4713e7f51fdbc33ed5746c83a9f6: Status 404 returned error can't find the container with id d7cddc583947a936caa034d25d3d79dcbb0b4713e7f51fdbc33ed5746c83a9f6 Nov 03 22:38:03.959557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:03.959517 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" event={"ID":"b8f6af3c-e6b8-412e-afad-296da72a59c7","Type":"ContainerStarted","Data":"c8dcaed2faeb2ceabb8d04452d3ab56a572bcc1263a377085510516346bd61a1"} Nov 03 22:38:03.959755 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:03.959616 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" event={"ID":"b8f6af3c-e6b8-412e-afad-296da72a59c7","Type":"ContainerStarted","Data":"d7cddc583947a936caa034d25d3d79dcbb0b4713e7f51fdbc33ed5746c83a9f6"} Nov 03 22:38:04.010253 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.010221 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:38:04.023663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.023642 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:38:04.070162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.070111 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" podStartSLOduration=2.070094647 podStartE2EDuration="2.070094647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:04.067850981 +0000 UTC m=+864.690159290" watchObservedRunningTime="2026-11-03 22:38:04.070094647 +0000 UTC m=+864.692402954" Nov 03 22:38:04.548428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.548395 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd"] Nov 03 22:38:04.552537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.552518 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.617107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.617082 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.617270 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.617117 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.617317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.617286 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftz2h\" (UniqueName: \"kubernetes.io/projected/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-kube-api-access-ftz2h\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.642064 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.642034 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd"] Nov 03 22:38:04.718275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.718243 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.718275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.718280 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.718475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.718322 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftz2h\" (UniqueName: \"kubernetes.io/projected/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-kube-api-access-ftz2h\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.718943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.718923 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.720604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.720586 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.775632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.775601 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftz2h\" (UniqueName: \"kubernetes.io/projected/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-kube-api-access-ftz2h\") pod \"maas-api-revoke-sub-1bc8efed-t2dsd\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.866102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.866017 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:04.969927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.969885 5272 generic.go:356] "Generic (PLEG): container finished" podID="b8f6af3c-e6b8-412e-afad-296da72a59c7" containerID="c8dcaed2faeb2ceabb8d04452d3ab56a572bcc1263a377085510516346bd61a1" exitCode=0 Nov 03 22:38:04.970083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.969978 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" event={"ID":"b8f6af3c-e6b8-412e-afad-296da72a59c7","Type":"ContainerDied","Data":"c8dcaed2faeb2ceabb8d04452d3ab56a572bcc1263a377085510516346bd61a1"} Nov 03 22:38:04.985129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:04.985100 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:38:05.240935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:05.240841 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd"] Nov 03 22:38:05.247778 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:38:05.247740 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d85d05_9c2c_4c59_af24_b3e1cc67ec53.slice/crio-623a5f3ce864238e3ef0f9389e528e75aa5831c48fcdbdfa5373fed45225cf30 WatchSource:0}: Error finding container 623a5f3ce864238e3ef0f9389e528e75aa5831c48fcdbdfa5373fed45225cf30: Status 404 returned error can't find the container with id 623a5f3ce864238e3ef0f9389e528e75aa5831c48fcdbdfa5373fed45225cf30 Nov 03 22:38:05.978358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:05.978257 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" event={"ID":"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53","Type":"ContainerStarted","Data":"5c7de86600070a44a57ef990dfb3953d0057cabd88d5de8ebe83945d0021fd4c"} Nov 03 22:38:05.978358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:05.978316 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" event={"ID":"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53","Type":"ContainerStarted","Data":"623a5f3ce864238e3ef0f9389e528e75aa5831c48fcdbdfa5373fed45225cf30"} Nov 03 22:38:06.089049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.088991 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" podStartSLOduration=2.088974842 podStartE2EDuration="2.088974842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:06.087879177 +0000 UTC m=+866.710187487" watchObservedRunningTime="2026-11-03 22:38:06.088974842 +0000 UTC m=+866.711283150" Nov 03 22:38:06.113617 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.113594 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:06.236342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.236254 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-xks-ca\") pod \"b8f6af3c-e6b8-412e-afad-296da72a59c7\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " Nov 03 22:38:06.236483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.236354 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-openshift-ca\") pod \"b8f6af3c-e6b8-412e-afad-296da72a59c7\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " Nov 03 22:38:06.236483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.236438 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f6af3c-e6b8-412e-afad-296da72a59c7-kube-api-access-78hbp\" (UniqueName: \"kubernetes.io/projected/b8f6af3c-e6b8-412e-afad-296da72a59c7-kube-api-access-78hbp\") pod \"b8f6af3c-e6b8-412e-afad-296da72a59c7\" (UID: \"b8f6af3c-e6b8-412e-afad-296da72a59c7\") " Nov 03 22:38:06.236745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.236712 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-openshift-ca" pod "b8f6af3c-e6b8-412e-afad-296da72a59c7" (UID: "b8f6af3c-e6b8-412e-afad-296da72a59c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:06.236918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.236900 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:06.238504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.238478 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-xks-ca" pod "b8f6af3c-e6b8-412e-afad-296da72a59c7" (UID: "b8f6af3c-e6b8-412e-afad-296da72a59c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:06.239033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.239014 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f6af3c-e6b8-412e-afad-296da72a59c7-kube-api-access-78hbp" pod "b8f6af3c-e6b8-412e-afad-296da72a59c7" (UID: "b8f6af3c-e6b8-412e-afad-296da72a59c7"). InnerVolumeSpecName "kube-api-access-78hbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:06.337741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.337702 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78hbp\" (UniqueName: \"kubernetes.io/projected/b8f6af3c-e6b8-412e-afad-296da72a59c7-kube-api-access-78hbp\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:06.337741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.337740 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f6af3c-e6b8-412e-afad-296da72a59c7-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:06.987348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.987314 5272 generic.go:356] "Generic (PLEG): container finished" podID="a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" containerID="5c7de86600070a44a57ef990dfb3953d0057cabd88d5de8ebe83945d0021fd4c" exitCode=0 Nov 03 22:38:06.987876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.987395 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" event={"ID":"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53","Type":"ContainerDied","Data":"5c7de86600070a44a57ef990dfb3953d0057cabd88d5de8ebe83945d0021fd4c"} Nov 03 22:38:06.989092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.989067 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" Nov 03 22:38:06.989209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.989117 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk" event={"ID":"b8f6af3c-e6b8-412e-afad-296da72a59c7","Type":"ContainerDied","Data":"d7cddc583947a936caa034d25d3d79dcbb0b4713e7f51fdbc33ed5746c83a9f6"} Nov 03 22:38:06.989209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:06.989162 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7cddc583947a936caa034d25d3d79dcbb0b4713e7f51fdbc33ed5746c83a9f6" Nov 03 22:38:08.248619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.248585 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:08.368535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.368494 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-openshift-ca\") pod \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " Nov 03 22:38:08.368737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.368580 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-kube-api-access-ftz2h\" (UniqueName: \"kubernetes.io/projected/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-kube-api-access-ftz2h\") pod \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " Nov 03 22:38:08.368737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.368712 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-xks-ca\") pod \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\" (UID: \"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53\") " Nov 03 22:38:08.368896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.368867 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-openshift-ca" pod "a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" (UID: "a4d85d05-9c2c-4c59-af24-b3e1cc67ec53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:08.369085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.369067 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:08.370774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.370745 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-xks-ca" pod "a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" (UID: "a4d85d05-9c2c-4c59-af24-b3e1cc67ec53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:08.370922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.370780 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-kube-api-access-ftz2h" pod "a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" (UID: "a4d85d05-9c2c-4c59-af24-b3e1cc67ec53"). InnerVolumeSpecName "kube-api-access-ftz2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:08.470380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.470342 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:08.470380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:08.470373 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftz2h\" (UniqueName: \"kubernetes.io/projected/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53-kube-api-access-ftz2h\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:09.024052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:09.024020 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" event={"ID":"a4d85d05-9c2c-4c59-af24-b3e1cc67ec53","Type":"ContainerDied","Data":"623a5f3ce864238e3ef0f9389e528e75aa5831c48fcdbdfa5373fed45225cf30"} Nov 03 22:38:09.024052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:09.024040 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd" Nov 03 22:38:09.024280 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:09.024068 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="623a5f3ce864238e3ef0f9389e528e75aa5831c48fcdbdfa5373fed45225cf30" Nov 03 22:38:17.779217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.779180 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz"] Nov 03 22:38:17.779746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.779730 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b8f6af3c-e6b8-412e-afad-296da72a59c7" containerName="revoke-keys" Nov 03 22:38:17.779836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.779748 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f6af3c-e6b8-412e-afad-296da72a59c7" containerName="revoke-keys" Nov 03 22:38:17.779836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.779786 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" containerName="revoke-keys" Nov 03 22:38:17.779836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.779797 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" containerName="revoke-keys" Nov 03 22:38:17.779939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.779908 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" containerName="revoke-keys" Nov 03 22:38:17.779939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.779923 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f6af3c-e6b8-412e-afad-296da72a59c7" containerName="revoke-keys" Nov 03 22:38:17.784955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.784936 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.837699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.837663 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz"] Nov 03 22:38:17.880230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.880186 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:17.880230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.880186 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:38:17.880722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.880693 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.880864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.880828 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.880864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.880791 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:38:17.880957 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.880908 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:38:17.880957 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.880939 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pnrl\" (UniqueName: \"kubernetes.io/projected/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-kube-api-access-8pnrl\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.981820 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.981780 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.982020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.981892 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pnrl\" (UniqueName: \"kubernetes.io/projected/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-kube-api-access-8pnrl\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.982020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.981947 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.982513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.982485 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:17.984496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:17.984474 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:18.039433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:18.039400 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pnrl\" (UniqueName: \"kubernetes.io/projected/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-kube-api-access-8pnrl\") pod \"maas-api-revoke-sub-5812b5d5-m5pzz\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:18.098373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:18.098341 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:18.475684 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:18.475583 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz"] Nov 03 22:38:18.482720 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:38:18.482690 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73c8102_dfd3_46cd_82cc_b7bb1b31e99e.slice/crio-034ee8918b351d56ad59c8bb5cf7eb7d0df36e979f2738058a22455c473cdee9 WatchSource:0}: Error finding container 034ee8918b351d56ad59c8bb5cf7eb7d0df36e979f2738058a22455c473cdee9: Status 404 returned error can't find the container with id 034ee8918b351d56ad59c8bb5cf7eb7d0df36e979f2738058a22455c473cdee9 Nov 03 22:38:19.088672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:19.088619 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" event={"ID":"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e","Type":"ContainerStarted","Data":"bc06d225ec088fccc8cd6ea2852a11fa8ca10a4058393e15feed2a0adda636fe"} Nov 03 22:38:19.089171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:19.088701 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" event={"ID":"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e","Type":"ContainerStarted","Data":"034ee8918b351d56ad59c8bb5cf7eb7d0df36e979f2738058a22455c473cdee9"} Nov 03 22:38:19.201788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:19.201717 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" podStartSLOduration=2.201699968 podStartE2EDuration="2.201699968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:19.198982172 +0000 UTC m=+879.821290480" watchObservedRunningTime="2026-11-03 22:38:19.201699968 +0000 UTC m=+879.824008279" Nov 03 22:38:20.049200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.049165 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p"] Nov 03 22:38:20.053389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.053366 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.096920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.096886 5272 generic.go:356] "Generic (PLEG): container finished" podID="d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" containerID="bc06d225ec088fccc8cd6ea2852a11fa8ca10a4058393e15feed2a0adda636fe" exitCode=0 Nov 03 22:38:20.097268 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.096966 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" event={"ID":"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e","Type":"ContainerDied","Data":"bc06d225ec088fccc8cd6ea2852a11fa8ca10a4058393e15feed2a0adda636fe"} Nov 03 22:38:20.100392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.100367 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56rx\" (UniqueName: \"kubernetes.io/projected/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-kube-api-access-c56rx\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.100512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.100484 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.100569 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.100531 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.108373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.108051 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p"] Nov 03 22:38:20.201834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.201775 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.202049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.201866 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.202049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.202003 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56rx\" (UniqueName: \"kubernetes.io/projected/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-kube-api-access-c56rx\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.202539 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.202512 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.204371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.204352 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.259230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.259203 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56rx\" (UniqueName: \"kubernetes.io/projected/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-kube-api-access-c56rx\") pod \"maas-api-revoke-sub-8c333ad2-h758p\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.366426 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.366339 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:20.740278 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:20.740179 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p"] Nov 03 22:38:20.745967 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:38:20.745937 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7cdb9e_c2e0_4ae9_9b13_239dc8a86423.slice/crio-718a4ea777132bcfc6fbd5c697ef6729cc51f89d035d97c9892a05e5794b17e9 WatchSource:0}: Error finding container 718a4ea777132bcfc6fbd5c697ef6729cc51f89d035d97c9892a05e5794b17e9: Status 404 returned error can't find the container with id 718a4ea777132bcfc6fbd5c697ef6729cc51f89d035d97c9892a05e5794b17e9 Nov 03 22:38:21.104692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.104660 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" event={"ID":"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423","Type":"ContainerStarted","Data":"718a4ea777132bcfc6fbd5c697ef6729cc51f89d035d97c9892a05e5794b17e9"} Nov 03 22:38:21.234445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.234421 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:21.313540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.313436 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-kube-api-access-8pnrl\" (UniqueName: \"kubernetes.io/projected/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-kube-api-access-8pnrl\") pod \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " Nov 03 22:38:21.313686 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.313569 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-openshift-ca\") pod \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " Nov 03 22:38:21.313686 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.313614 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-xks-ca\") pod \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\" (UID: \"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e\") " Nov 03 22:38:21.313992 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.313967 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-openshift-ca" pod "d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" (UID: "d73c8102-dfd3-46cd-82cc-b7bb1b31e99e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:21.315713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.315690 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-kube-api-access-8pnrl" pod "d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" (UID: "d73c8102-dfd3-46cd-82cc-b7bb1b31e99e"). InnerVolumeSpecName "kube-api-access-8pnrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:21.315780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.315710 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-xks-ca" pod "d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" (UID: "d73c8102-dfd3-46cd-82cc-b7bb1b31e99e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:21.414728 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.414694 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pnrl\" (UniqueName: \"kubernetes.io/projected/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-kube-api-access-8pnrl\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:21.414728 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.414725 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:21.414951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:21.414739 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:22.111732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:22.111696 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" Nov 03 22:38:22.111732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:22.111714 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz" event={"ID":"d73c8102-dfd3-46cd-82cc-b7bb1b31e99e","Type":"ContainerDied","Data":"034ee8918b351d56ad59c8bb5cf7eb7d0df36e979f2738058a22455c473cdee9"} Nov 03 22:38:22.112274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:22.111815 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="034ee8918b351d56ad59c8bb5cf7eb7d0df36e979f2738058a22455c473cdee9" Nov 03 22:38:22.117360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:22.117327 5272 generic.go:356] "Generic (PLEG): container finished" podID="1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" containerID="4576a026b45d9153eaae7fdd32cc84300d00e02a514cc7213b742bcd4f80ea23" exitCode=0 Nov 03 22:38:22.117503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:22.117377 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" event={"ID":"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423","Type":"ContainerDied","Data":"4576a026b45d9153eaae7fdd32cc84300d00e02a514cc7213b742bcd4f80ea23"} Nov 03 22:38:23.266595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.266572 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:23.335927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.335892 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-kube-api-access-c56rx\" (UniqueName: \"kubernetes.io/projected/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-kube-api-access-c56rx\") pod \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " Nov 03 22:38:23.336106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.335953 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-xks-ca\") pod \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " Nov 03 22:38:23.336106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.336022 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-openshift-ca\") pod \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\" (UID: \"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423\") " Nov 03 22:38:23.336417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.336389 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-openshift-ca" pod "1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" (UID: "1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:23.338080 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.338060 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-xks-ca" pod "1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" (UID: "1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:23.338186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.338167 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-kube-api-access-c56rx" pod "1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" (UID: "1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423"). InnerVolumeSpecName "kube-api-access-c56rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:23.437274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.437185 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c56rx\" (UniqueName: \"kubernetes.io/projected/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-kube-api-access-c56rx\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:23.437274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.437222 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:23.437274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:23.437237 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:24.132593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:24.132548 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" event={"ID":"1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423","Type":"ContainerDied","Data":"718a4ea777132bcfc6fbd5c697ef6729cc51f89d035d97c9892a05e5794b17e9"} Nov 03 22:38:24.132593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:24.132574 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p" Nov 03 22:38:24.132904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:24.132605 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718a4ea777132bcfc6fbd5c697ef6729cc51f89d035d97c9892a05e5794b17e9" Nov 03 22:38:24.500283 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:38:24.500196 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:38:25.721842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:25.721798 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:38:32.568954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.568920 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl"] Nov 03 22:38:32.569752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.569525 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" containerName="revoke-keys" Nov 03 22:38:32.569752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.569541 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" containerName="revoke-keys" Nov 03 22:38:32.569752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.569550 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" containerName="revoke-keys" Nov 03 22:38:32.569752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.569555 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" containerName="revoke-keys" Nov 03 22:38:32.569752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.569649 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" containerName="revoke-keys" Nov 03 22:38:32.569752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.569658 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" containerName="revoke-keys" Nov 03 22:38:32.575041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.575021 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645614 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645816 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645677 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645816 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645751 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645829 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645864 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645912 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645941 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.645963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.645960 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49579\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-kube-api-access-49579\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.646078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.646025 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.672615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.672582 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-68fbd1-openshift-default-dockercfg-sd2d7\"" Nov 03 22:38:32.675848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.675826 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl"] Nov 03 22:38:32.747172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747142 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747180 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747197 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747231 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747338 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747384 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747426 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747455 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49579\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-kube-api-access-49579\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747773 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747846 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.747902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747887 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.748010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747776 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.748010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.747985 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.748163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.748142 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.749500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.749477 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.749793 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.749776 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.807678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.807648 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.807833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.807715 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49579\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-kube-api-access-49579\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:32.887975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:32.887901 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:33.318012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:33.317975 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl"] Nov 03 22:38:33.323725 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:38:33.323698 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecd2d08_ed0c_401e_a875_a9d7bf75b4cf.slice/crio-865bc2100a99affa8a35b1f7d385d6e1c808256f0e2b23a874821e0198de4f35 WatchSource:0}: Error finding container 865bc2100a99affa8a35b1f7d385d6e1c808256f0e2b23a874821e0198de4f35: Status 404 returned error can't find the container with id 865bc2100a99affa8a35b1f7d385d6e1c808256f0e2b23a874821e0198de4f35 Nov 03 22:38:33.326182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:33.326145 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:38:33.326305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:33.326228 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:38:33.326305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:33.326273 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:38:34.194730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:34.194688 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" event={"ID":"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf","Type":"ContainerStarted","Data":"bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c"} Nov 03 22:38:34.195254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:34.194750 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" event={"ID":"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf","Type":"ContainerStarted","Data":"865bc2100a99affa8a35b1f7d385d6e1c808256f0e2b23a874821e0198de4f35"} Nov 03 22:38:34.888375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:34.888340 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:34.889770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:34.889749 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:38:34.889846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:34.889826 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:38:35.889269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:35.889234 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:38:35.889673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:35.889297 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:38:36.888489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:36.888449 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:38:36.888710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:36.888526 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:38:37.889261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:37.889222 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:38:37.889694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:37.889286 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:38:38.889280 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:38.889240 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:38:38.889786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:38.889308 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:38:39.888760 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:39.888716 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:38:39.888954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:39.888785 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:38:40.893543 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:40.893505 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:40.894040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:40.894023 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:40.894817 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:40.894780 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:38:41.056575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:41.056517 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podStartSLOduration=9.056503232 podStartE2EDuration="9.056503232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:34.421156622 +0000 UTC m=+895.043464929" watchObservedRunningTime="2026-11-03 22:38:41.056503232 +0000 UTC m=+901.678811540" Nov 03 22:38:42.994686 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:42.994650 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q"] Nov 03 22:38:43.000085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.000060 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.053219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.053187 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q"] Nov 03 22:38:43.072635 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.072602 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-dshm\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.072788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.072713 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-model-cache\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.072788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.072774 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tls-certs\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.072892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.072825 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-home\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.072892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.072856 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tmp-dir\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.073014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.072989 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rgts\" (UniqueName: \"kubernetes.io/projected/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-kube-api-access-4rgts\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.095699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.095675 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-961054\"/\"distinct-w3-961054-kserve-self-signed-certs\"" Nov 03 22:38:43.174597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.174561 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tls-certs\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.174771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.174610 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-home\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.174771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.174730 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tmp-dir\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.174892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.174818 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rgts\" (UniqueName: \"kubernetes.io/projected/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-kube-api-access-4rgts\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.174937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.174911 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-dshm\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.174987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.174977 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-model-cache\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.175043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.175006 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-home\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.175094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.175052 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tmp-dir\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.175304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.175285 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-model-cache\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.176962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.176938 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-dshm\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.177150 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.177134 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tls-certs\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.232353 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.232320 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rgts\" (UniqueName: \"kubernetes.io/projected/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-kube-api-access-4rgts\") pod \"distinct-w3-961054-kserve-65d7477cd5-dcp2q\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.311833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.311777 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:43.741464 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:43.741378 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q"] Nov 03 22:38:43.746526 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:38:43.746496 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cb4b4c2_02ad_468e_ba1a_7625d94a2f55.slice/crio-cdb936f620b7ad139c5849f18cd4e9ee57babe356441baa562f9310c1f0354e1 WatchSource:0}: Error finding container cdb936f620b7ad139c5849f18cd4e9ee57babe356441baa562f9310c1f0354e1: Status 404 returned error can't find the container with id cdb936f620b7ad139c5849f18cd4e9ee57babe356441baa562f9310c1f0354e1 Nov 03 22:38:44.261915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:44.261878 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" event={"ID":"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55","Type":"ContainerStarted","Data":"07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0"} Nov 03 22:38:44.261915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:44.261921 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" event={"ID":"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55","Type":"ContainerStarted","Data":"cdb936f620b7ad139c5849f18cd4e9ee57babe356441baa562f9310c1f0354e1"} Nov 03 22:38:44.312676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:44.312641 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:44.372921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:44.372861 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podStartSLOduration=2.372848528 podStartE2EDuration="2.372848528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:44.370837225 +0000 UTC m=+904.993145532" watchObservedRunningTime="2026-11-03 22:38:44.372848528 +0000 UTC m=+904.995156891" Nov 03 22:38:46.718117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.718080 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb"] Nov 03 22:38:46.722468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.722447 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.819781 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.819747 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ee74cb\"/\"distinct-w2-ee74cb-kserve-self-signed-certs\"" Nov 03 22:38:46.823279 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.823254 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb"] Nov 03 22:38:46.830372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.830354 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-dshm\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.830429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.830386 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6d2v\" (UniqueName: \"kubernetes.io/projected/a5042eed-9361-438a-a670-8b913eeac140-kube-api-access-t6d2v\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.830500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.830484 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-model-cache\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.830612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.830592 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-tmp-dir\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.830649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.830626 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-home\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.830723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.830709 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5042eed-9361-438a-a670-8b913eeac140-tls-certs\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.931706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.931669 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-dshm\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.931706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.931711 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6d2v\" (UniqueName: \"kubernetes.io/projected/a5042eed-9361-438a-a670-8b913eeac140-kube-api-access-t6d2v\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.931986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.931753 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-model-cache\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.931986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.931850 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-tmp-dir\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.931986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.931872 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-home\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.931986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.931910 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5042eed-9361-438a-a670-8b913eeac140-tls-certs\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.932295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.932261 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-model-cache\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.932408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.932276 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-tmp-dir\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.932408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.932333 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-home\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.934031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.934005 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-dshm\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.934401 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.934384 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5042eed-9361-438a-a670-8b913eeac140-tls-certs\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:46.989400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:46.989327 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6d2v\" (UniqueName: \"kubernetes.io/projected/a5042eed-9361-438a-a670-8b913eeac140-kube-api-access-t6d2v\") pod \"distinct-w2-ee74cb-kserve-64f486db99-9zlvb\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:47.035335 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:47.035295 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:47.451841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:47.451789 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb"] Nov 03 22:38:47.457150 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:38:47.457114 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5042eed_9361_438a_a670_8b913eeac140.slice/crio-3c36e2288b91c5316fac3eaf7962383a80062c796c8ec791abd3fb3fa760ffd7 WatchSource:0}: Error finding container 3c36e2288b91c5316fac3eaf7962383a80062c796c8ec791abd3fb3fa760ffd7: Status 404 returned error can't find the container with id 3c36e2288b91c5316fac3eaf7962383a80062c796c8ec791abd3fb3fa760ffd7 Nov 03 22:38:48.317065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:48.317028 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" event={"ID":"a5042eed-9361-438a-a670-8b913eeac140","Type":"ContainerStarted","Data":"a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804"} Nov 03 22:38:48.317481 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:48.317075 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" event={"ID":"a5042eed-9361-438a-a670-8b913eeac140","Type":"ContainerStarted","Data":"3c36e2288b91c5316fac3eaf7962383a80062c796c8ec791abd3fb3fa760ffd7"} Nov 03 22:38:48.428526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:48.428466 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podStartSLOduration=2.42845187 podStartE2EDuration="2.42845187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:48.426295625 +0000 UTC m=+909.048603933" watchObservedRunningTime="2026-11-03 22:38:48.42845187 +0000 UTC m=+909.050760180" Nov 03 22:38:49.036302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:49.036264 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:53.312517 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:53.312478 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:53.325204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:53.325181 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:53.359361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:53.359333 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:38:54.501211 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:38:54.501166 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:38:57.036273 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:57.036234 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:57.049308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:57.049274 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:57.412302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:57.412201 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:38:57.622110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:38:57.622075 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:39:12.882778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.882740 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv"] Nov 03 22:39:12.887342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.887317 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:12.943523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.943486 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-home\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:12.943523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.943546 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-dshm\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:12.943837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.943698 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-tmp-dir\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:12.943837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.943751 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b720de-005d-4af7-b356-2d2387bf4e57-tls-certs\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:12.943837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.943832 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-model-cache\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:12.944020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.943860 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22p7k\" (UniqueName: \"kubernetes.io/projected/f3b720de-005d-4af7-b356-2d2387bf4e57-kube-api-access-22p7k\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:12.984070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.984034 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5e72ad\"/\"distinct-w6-5e72ad-kserve-self-signed-certs\"" Nov 03 22:39:12.988609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:12.988571 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv"] Nov 03 22:39:13.044701 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.044658 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-tmp-dir\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.049959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.045628 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-tmp-dir\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.052773 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.052740 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b720de-005d-4af7-b356-2d2387bf4e57-tls-certs\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.053096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.053071 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-model-cache\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.056432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.056400 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22p7k\" (UniqueName: \"kubernetes.io/projected/f3b720de-005d-4af7-b356-2d2387bf4e57-kube-api-access-22p7k\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.057003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.056339 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-model-cache\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.058200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.058175 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-home\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.058337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.058320 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-dshm\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.059403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.058542 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-home\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.062272 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.062249 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-dshm\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.066200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.066171 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b720de-005d-4af7-b356-2d2387bf4e57-tls-certs\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.114930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.114885 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22p7k\" (UniqueName: \"kubernetes.io/projected/f3b720de-005d-4af7-b356-2d2387bf4e57-kube-api-access-22p7k\") pod \"distinct-w6-5e72ad-kserve-58f549d47c-q2lsv\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.201406 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.201309 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:13.629190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:13.629159 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv"] Nov 03 22:39:13.634848 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:39:13.634792 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b720de_005d_4af7_b356_2d2387bf4e57.slice/crio-f97ee13b30cea1005655e136c4240f127a3aadaf55e7f81d7b1a0634251bc2d0 WatchSource:0}: Error finding container f97ee13b30cea1005655e136c4240f127a3aadaf55e7f81d7b1a0634251bc2d0: Status 404 returned error can't find the container with id f97ee13b30cea1005655e136c4240f127a3aadaf55e7f81d7b1a0634251bc2d0 Nov 03 22:39:14.543967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:14.543925 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" event={"ID":"f3b720de-005d-4af7-b356-2d2387bf4e57","Type":"ContainerStarted","Data":"7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989"} Nov 03 22:39:14.543967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:14.543971 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" event={"ID":"f3b720de-005d-4af7-b356-2d2387bf4e57","Type":"ContainerStarted","Data":"f97ee13b30cea1005655e136c4240f127a3aadaf55e7f81d7b1a0634251bc2d0"} Nov 03 22:39:14.654270 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:14.654211 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podStartSLOduration=2.654194473 podStartE2EDuration="2.654194473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:14.653483676 +0000 UTC m=+935.275791984" watchObservedRunningTime="2026-11-03 22:39:14.654194473 +0000 UTC m=+935.276502780" Nov 03 22:39:15.202421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:15.202381 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:23.201667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:23.201628 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:23.214439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:23.214414 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:23.611361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:23.611334 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:39:24.499450 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:39:24.499415 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:39:28.396918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:28.396877 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:39:47.458045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.458007 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt"] Nov 03 22:39:47.463620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.463596 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.557567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.557528 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.557753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.557617 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.557753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.557643 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.557753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.557742 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.557954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.557763 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8k4\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-kube-api-access-kq8k4\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.557954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.557883 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.557954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.557912 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829cd7a-0856-4739-a116-68de1211fb5b-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.558100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.558009 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829cd7a-0856-4739-a116-68de1211fb5b-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.558100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.558052 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.560583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.560547 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5e72ad-openshift-default-dockercfg-5qnwn\"" Nov 03 22:39:47.565384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.565350 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt"] Nov 03 22:39:47.659048 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659013 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829cd7a-0856-4739-a116-68de1211fb5b-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659058 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659180 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659223 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659246 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659498 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659324 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659498 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659355 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8k4\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-kube-api-access-kq8k4\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659498 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659419 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659498 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659453 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829cd7a-0856-4739-a116-68de1211fb5b-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659614 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-credential-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659656 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-socket\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.659959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659937 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-certs\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.660065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.659971 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-data\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.660273 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.660251 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829cd7a-0856-4739-a116-68de1211fb5b-istiod-ca-cert\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.661743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.661721 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829cd7a-0856-4739-a116-68de1211fb5b-istio-podinfo\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.661854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.661775 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-envoy\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.716900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.716826 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-istio-token\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.717041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.716915 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8k4\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-kube-api-access-kq8k4\") pod \"e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:47.777378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:47.777337 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:48.211071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.211034 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt"] Nov 03 22:39:48.216046 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:39:48.216017 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9829cd7a_0856_4739_a116_68de1211fb5b.slice/crio-5fc894c59636d62d1c6715500034363fd86857b84d33968c0a68345b87d89573 WatchSource:0}: Error finding container 5fc894c59636d62d1c6715500034363fd86857b84d33968c0a68345b87d89573: Status 404 returned error can't find the container with id 5fc894c59636d62d1c6715500034363fd86857b84d33968c0a68345b87d89573 Nov 03 22:39:48.218428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.218395 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:48.218488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.218463 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:48.218528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.218493 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:39:48.767844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.767793 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" event={"ID":"9829cd7a-0856-4739-a116-68de1211fb5b","Type":"ContainerStarted","Data":"b3314333f7f73292a97824e49a536e040ec329e998a7168cb9bb9840ac0af972"} Nov 03 22:39:48.768244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.767862 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" event={"ID":"9829cd7a-0856-4739-a116-68de1211fb5b","Type":"ContainerStarted","Data":"5fc894c59636d62d1c6715500034363fd86857b84d33968c0a68345b87d89573"} Nov 03 22:39:48.778251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.778226 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:48.883379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:48.883319 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podStartSLOduration=1.8833057549999999 podStartE2EDuration="1.883305755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:48.881543779 +0000 UTC m=+969.503852110" watchObservedRunningTime="2026-11-03 22:39:48.883305755 +0000 UTC m=+969.505614062" Nov 03 22:39:49.779421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:49.779389 5272 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:39:49.779833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:49.779443 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 22:39:50.778516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:50.778479 5272 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:39:50.778723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:50.778541 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 22:39:51.778064 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:51.778022 5272 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:39:51.778573 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:51.778091 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 22:39:52.778114 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:52.778078 5272 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:39:52.778565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:52.778136 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 22:39:53.778008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:53.777967 5272 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:39:53.778196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:53.778026 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 22:39:54.500880 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:39:54.500843 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:39:54.778106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:54.778001 5272 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:39:54.778106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:54.778058 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 22:39:55.782453 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:55.782423 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:55.782971 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:55.782793 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:39:55.783499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:39:55.783478 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 22:40:07.564159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:07.564123 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:40:17.989332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:17.989296 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc"] Nov 03 22:40:17.993743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:17.993722 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090285 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090515 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090584 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090651 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090676 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090727 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjrj\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-kube-api-access-bdjrj\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090776 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090829 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.090862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.090857 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.093594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.093573 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc"] Nov 03 22:40:18.192099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192064 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192122 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192198 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192222 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192260 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192277 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192330 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjrj\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-kube-api-access-bdjrj\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192381 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192405 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.192951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192895 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.193105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.192954 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.193105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.193009 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.193239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.193146 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.193239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.193201 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.194869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.194834 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.196205 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.196187 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.272926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.271987 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.286384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.286309 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjrj\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-kube-api-access-bdjrj\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.308141 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.308111 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:18.745739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.745705 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc"] Nov 03 22:40:18.750997 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:40:18.750965 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e1f824_3797_4a06_815e_bc5ef21b3f2b.slice/crio-019c3bc724f9e13b41a57093a7763a4799faf3b072606058e6c5862804efd839 WatchSource:0}: Error finding container 019c3bc724f9e13b41a57093a7763a4799faf3b072606058e6c5862804efd839: Status 404 returned error can't find the container with id 019c3bc724f9e13b41a57093a7763a4799faf3b072606058e6c5862804efd839 Nov 03 22:40:18.753299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.753262 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:40:18.753391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.753333 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:40:18.753391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.753363 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:40:18.975667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.975626 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" event={"ID":"66e1f824-3797-4a06-815e-bc5ef21b3f2b","Type":"ContainerStarted","Data":"edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6"} Nov 03 22:40:18.975876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:18.975694 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" event={"ID":"66e1f824-3797-4a06-815e-bc5ef21b3f2b","Type":"ContainerStarted","Data":"019c3bc724f9e13b41a57093a7763a4799faf3b072606058e6c5862804efd839"} Nov 03 22:40:19.090227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:19.090178 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podStartSLOduration=2.090163936 podStartE2EDuration="2.090163936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:19.088121606 +0000 UTC m=+999.710429927" watchObservedRunningTime="2026-11-03 22:40:19.090163936 +0000 UTC m=+999.712472244" Nov 03 22:40:19.308620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:19.308584 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:19.309994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:19.309967 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:40:19.310081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:19.310030 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 22:40:20.308620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:20.308531 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:40:20.308620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:20.308591 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 22:40:21.308959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:21.308919 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:40:21.309382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:21.308977 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 22:40:22.309380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:22.309347 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:40:22.309899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:22.309408 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 22:40:23.309003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:23.308968 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:40:23.309194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:23.309027 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 22:40:24.309128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:24.309076 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:40:24.309618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:24.309146 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 22:40:24.501490 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:40:24.501453 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:40:25.309366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:25.309331 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 22:40:25.309786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:25.309398 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 22:40:26.314106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:26.314071 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:26.314571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:26.314545 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:26.315313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:26.315293 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:40:30.651004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.650969 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp"] Nov 03 22:40:30.655264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.655242 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.705353 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.705272 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:40:30.705353 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.705335 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:40:30.705617 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.705354 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:40:30.751813 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.751766 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:40:30.755332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.755308 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp"] Nov 03 22:40:30.771738 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.771705 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqgw\" (UniqueName: \"kubernetes.io/projected/8f8fcc5b-a162-440f-b537-77bf5f566a85-kube-api-access-mmqgw\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.771907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.771742 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.771977 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.771956 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.872916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.872881 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqgw\" (UniqueName: \"kubernetes.io/projected/8f8fcc5b-a162-440f-b537-77bf5f566a85-kube-api-access-mmqgw\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.872916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.872916 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.873241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.872988 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.873605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.873580 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.875303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.875284 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.930479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.930408 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqgw\" (UniqueName: \"kubernetes.io/projected/8f8fcc5b-a162-440f-b537-77bf5f566a85-kube-api-access-mmqgw\") pod \"maas-api-revoke-sub-3c62007c-qbggp\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:30.969992 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:30.969960 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:31.411365 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:31.411329 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp"] Nov 03 22:40:31.416868 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:40:31.416838 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8fcc5b_a162_440f_b537_77bf5f566a85.slice/crio-1ef0b38586dae8cef173c0c34acff70d9a0d7f9b5bec4a5f3bc4e937fb0b2101 WatchSource:0}: Error finding container 1ef0b38586dae8cef173c0c34acff70d9a0d7f9b5bec4a5f3bc4e937fb0b2101: Status 404 returned error can't find the container with id 1ef0b38586dae8cef173c0c34acff70d9a0d7f9b5bec4a5f3bc4e937fb0b2101 Nov 03 22:40:32.075216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:32.075169 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" event={"ID":"8f8fcc5b-a162-440f-b537-77bf5f566a85","Type":"ContainerStarted","Data":"27a268226c3ce438901df4a340a74d9acdb7e2a302fd21afce0c30be5d4bd2f9"} Nov 03 22:40:32.075616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:32.075241 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" event={"ID":"8f8fcc5b-a162-440f-b537-77bf5f566a85","Type":"ContainerStarted","Data":"1ef0b38586dae8cef173c0c34acff70d9a0d7f9b5bec4a5f3bc4e937fb0b2101"} Nov 03 22:40:33.083866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:33.083824 5272 generic.go:356] "Generic (PLEG): container finished" podID="8f8fcc5b-a162-440f-b537-77bf5f566a85" containerID="27a268226c3ce438901df4a340a74d9acdb7e2a302fd21afce0c30be5d4bd2f9" exitCode=0 Nov 03 22:40:33.084332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:33.083900 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" event={"ID":"8f8fcc5b-a162-440f-b537-77bf5f566a85","Type":"ContainerDied","Data":"27a268226c3ce438901df4a340a74d9acdb7e2a302fd21afce0c30be5d4bd2f9"} Nov 03 22:40:34.226068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.226042 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:34.233559 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.233535 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-xks-ca\") pod \"8f8fcc5b-a162-440f-b537-77bf5f566a85\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " Nov 03 22:40:34.233683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.233599 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-openshift-ca\") pod \"8f8fcc5b-a162-440f-b537-77bf5f566a85\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " Nov 03 22:40:34.233725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.233698 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8fcc5b-a162-440f-b537-77bf5f566a85-kube-api-access-mmqgw\" (UniqueName: \"kubernetes.io/projected/8f8fcc5b-a162-440f-b537-77bf5f566a85-kube-api-access-mmqgw\") pod \"8f8fcc5b-a162-440f-b537-77bf5f566a85\" (UID: \"8f8fcc5b-a162-440f-b537-77bf5f566a85\") " Nov 03 22:40:34.233978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.233950 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-openshift-ca" pod "8f8fcc5b-a162-440f-b537-77bf5f566a85" (UID: "8f8fcc5b-a162-440f-b537-77bf5f566a85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:34.234083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.234045 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:34.235672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.235649 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-xks-ca" pod "8f8fcc5b-a162-440f-b537-77bf5f566a85" (UID: "8f8fcc5b-a162-440f-b537-77bf5f566a85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:34.235758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.235689 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8fcc5b-a162-440f-b537-77bf5f566a85-kube-api-access-mmqgw" pod "8f8fcc5b-a162-440f-b537-77bf5f566a85" (UID: "8f8fcc5b-a162-440f-b537-77bf5f566a85"). InnerVolumeSpecName "kube-api-access-mmqgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:34.335125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.335091 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmqgw\" (UniqueName: \"kubernetes.io/projected/8f8fcc5b-a162-440f-b537-77bf5f566a85-kube-api-access-mmqgw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:34.335125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:34.335123 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8fcc5b-a162-440f-b537-77bf5f566a85-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:35.098897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:35.098845 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" event={"ID":"8f8fcc5b-a162-440f-b537-77bf5f566a85","Type":"ContainerDied","Data":"1ef0b38586dae8cef173c0c34acff70d9a0d7f9b5bec4a5f3bc4e937fb0b2101"} Nov 03 22:40:35.099078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:35.098945 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ef0b38586dae8cef173c0c34acff70d9a0d7f9b5bec4a5f3bc4e937fb0b2101" Nov 03 22:40:35.099078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:35.098862 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp" Nov 03 22:40:44.859019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.858979 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf"] Nov 03 22:40:44.859537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.859513 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8f8fcc5b-a162-440f-b537-77bf5f566a85" containerName="revoke-keys" Nov 03 22:40:44.859537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.859527 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8fcc5b-a162-440f-b537-77bf5f566a85" containerName="revoke-keys" Nov 03 22:40:44.859637 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.859626 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8fcc5b-a162-440f-b537-77bf5f566a85" containerName="revoke-keys" Nov 03 22:40:44.863601 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.863576 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:44.917559 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.917526 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf"] Nov 03 22:40:44.959373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.959342 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:40:44.959561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.959399 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:40:44.959561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.959407 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:40:44.959561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.959405 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:40:44.964028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.964007 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:44.964088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.964039 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:44.964196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:44.964176 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zq8g\" (UniqueName: \"kubernetes.io/projected/a1fd1c4e-3d51-4757-8853-2a0370512b0f-kube-api-access-6zq8g\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.065733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.065689 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.065733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.065735 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.066045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.065867 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zq8g\" (UniqueName: \"kubernetes.io/projected/a1fd1c4e-3d51-4757-8853-2a0370512b0f-kube-api-access-6zq8g\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.066505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.066482 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.068147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.068122 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.123722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.123648 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zq8g\" (UniqueName: \"kubernetes.io/projected/a1fd1c4e-3d51-4757-8853-2a0370512b0f-kube-api-access-6zq8g\") pod \"maas-api-revoke-sub-42356dd9-cjwwf\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.177678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.177641 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:45.553667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:45.553623 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf"] Nov 03 22:40:45.559216 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:40:45.559186 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fd1c4e_3d51_4757_8853_2a0370512b0f.slice/crio-99129566c19bf4184c4cec71bd6e91cb5cca1e6d8b94f474748e2f2ed7b4e569 WatchSource:0}: Error finding container 99129566c19bf4184c4cec71bd6e91cb5cca1e6d8b94f474748e2f2ed7b4e569: Status 404 returned error can't find the container with id 99129566c19bf4184c4cec71bd6e91cb5cca1e6d8b94f474748e2f2ed7b4e569 Nov 03 22:40:46.170237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:46.170145 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" event={"ID":"a1fd1c4e-3d51-4757-8853-2a0370512b0f","Type":"ContainerStarted","Data":"450f423b60274b6b1bb43fdc00692a39c102d9cbea2c7c885337f77e5893056f"} Nov 03 22:40:46.170607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:46.170243 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" event={"ID":"a1fd1c4e-3d51-4757-8853-2a0370512b0f","Type":"ContainerStarted","Data":"99129566c19bf4184c4cec71bd6e91cb5cca1e6d8b94f474748e2f2ed7b4e569"} Nov 03 22:40:46.280250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:46.280196 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" podStartSLOduration=2.280182008 podStartE2EDuration="2.280182008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:46.278146183 +0000 UTC m=+1026.900454491" watchObservedRunningTime="2026-11-03 22:40:46.280182008 +0000 UTC m=+1026.902490317" Nov 03 22:40:47.178657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:47.178621 5272 generic.go:356] "Generic (PLEG): container finished" podID="a1fd1c4e-3d51-4757-8853-2a0370512b0f" containerID="450f423b60274b6b1bb43fdc00692a39c102d9cbea2c7c885337f77e5893056f" exitCode=0 Nov 03 22:40:47.179057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:47.178678 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" event={"ID":"a1fd1c4e-3d51-4757-8853-2a0370512b0f","Type":"ContainerDied","Data":"450f423b60274b6b1bb43fdc00692a39c102d9cbea2c7c885337f77e5893056f"} Nov 03 22:40:48.336862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.336831 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:48.408283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.408256 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-xks-ca\") pod \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " Nov 03 22:40:48.408513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.408391 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-openshift-ca\") pod \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " Nov 03 22:40:48.408593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.408533 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fd1c4e-3d51-4757-8853-2a0370512b0f-kube-api-access-6zq8g\" (UniqueName: \"kubernetes.io/projected/a1fd1c4e-3d51-4757-8853-2a0370512b0f-kube-api-access-6zq8g\") pod \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\" (UID: \"a1fd1c4e-3d51-4757-8853-2a0370512b0f\") " Nov 03 22:40:48.408824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.408779 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-openshift-ca" pod "a1fd1c4e-3d51-4757-8853-2a0370512b0f" (UID: "a1fd1c4e-3d51-4757-8853-2a0370512b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:48.409036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.409007 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:48.410628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.410602 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fd1c4e-3d51-4757-8853-2a0370512b0f-kube-api-access-6zq8g" pod "a1fd1c4e-3d51-4757-8853-2a0370512b0f" (UID: "a1fd1c4e-3d51-4757-8853-2a0370512b0f"). InnerVolumeSpecName "kube-api-access-6zq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:48.410878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.410857 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-xks-ca" pod "a1fd1c4e-3d51-4757-8853-2a0370512b0f" (UID: "a1fd1c4e-3d51-4757-8853-2a0370512b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:48.510361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.510271 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zq8g\" (UniqueName: \"kubernetes.io/projected/a1fd1c4e-3d51-4757-8853-2a0370512b0f-kube-api-access-6zq8g\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:48.510361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:48.510300 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fd1c4e-3d51-4757-8853-2a0370512b0f-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:49.194763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:49.194730 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" Nov 03 22:40:49.194947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:49.194758 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf" event={"ID":"a1fd1c4e-3d51-4757-8853-2a0370512b0f","Type":"ContainerDied","Data":"99129566c19bf4184c4cec71bd6e91cb5cca1e6d8b94f474748e2f2ed7b4e569"} Nov 03 22:40:49.194947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:49.194824 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99129566c19bf4184c4cec71bd6e91cb5cca1e6d8b94f474748e2f2ed7b4e569" Nov 03 22:40:52.298304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:40:52.298253 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:40:54.500414 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:40:54.500376 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:41:04.069886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:04.069847 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj"] Nov 03 22:41:04.115973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:04.115936 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b44a677-cq5kj"] Nov 03 22:41:05.493736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:05.493701 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cd0ebc-7ecc-44f3-9dc0-b128c68e628b" path="/var/lib/kubelet/pods/79cd0ebc-7ecc-44f3-9dc0-b128c68e628b/volumes" Nov 03 22:41:06.599843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.599795 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz"] Nov 03 22:41:06.600421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.600404 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a1fd1c4e-3d51-4757-8853-2a0370512b0f" containerName="revoke-keys" Nov 03 22:41:06.600487 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.600421 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1fd1c4e-3d51-4757-8853-2a0370512b0f" containerName="revoke-keys" Nov 03 22:41:06.600552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.600524 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1fd1c4e-3d51-4757-8853-2a0370512b0f" containerName="revoke-keys" Nov 03 22:41:06.604308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.604280 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.658902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.658855 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz"] Nov 03 22:41:06.701294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.701262 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ee74cb\"/\"distinct-2-w2-ee74cb-kserve-self-signed-certs\"" Nov 03 22:41:06.730990 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.730958 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-tmp-dir\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.731169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.731089 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e875ca-63d0-4219-af46-5247b25c98db-tls-certs\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.731169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.731145 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwh8\" (UniqueName: \"kubernetes.io/projected/17e875ca-63d0-4219-af46-5247b25c98db-kube-api-access-7rwh8\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.731329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.731175 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-home\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.731329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.731245 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-model-cache\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.731438 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.731332 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-dshm\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.831997 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwh8\" (UniqueName: \"kubernetes.io/projected/17e875ca-63d0-4219-af46-5247b25c98db-kube-api-access-7rwh8\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832039 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-home\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832086 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-model-cache\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832143 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-dshm\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832207 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-tmp-dir\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832276 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e875ca-63d0-4219-af46-5247b25c98db-tls-certs\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832548 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-home\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832591 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-model-cache\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.832718 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.832700 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-tmp-dir\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.834570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.834552 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-dshm\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.834654 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.834637 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e875ca-63d0-4219-af46-5247b25c98db-tls-certs\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.889509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.889434 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwh8\" (UniqueName: \"kubernetes.io/projected/17e875ca-63d0-4219-af46-5247b25c98db-kube-api-access-7rwh8\") pod \"distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:06.917001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:06.916971 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:07.365229 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:07.363932 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17e875ca_63d0_4219_af46_5247b25c98db.slice/crio-c894e516a8e816b25b526271b9db4ef6424690099a153f4997261d020e81fdc3 WatchSource:0}: Error finding container c894e516a8e816b25b526271b9db4ef6424690099a153f4997261d020e81fdc3: Status 404 returned error can't find the container with id c894e516a8e816b25b526271b9db4ef6424690099a153f4997261d020e81fdc3 Nov 03 22:41:07.365229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:07.364329 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz"] Nov 03 22:41:08.374593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:08.374557 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" event={"ID":"17e875ca-63d0-4219-af46-5247b25c98db","Type":"ContainerStarted","Data":"3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae"} Nov 03 22:41:08.375010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:08.374607 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" event={"ID":"17e875ca-63d0-4219-af46-5247b25c98db","Type":"ContainerStarted","Data":"c894e516a8e816b25b526271b9db4ef6424690099a153f4997261d020e81fdc3"} Nov 03 22:41:08.485149 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:08.485094 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podStartSLOduration=2.485078642 podStartE2EDuration="2.485078642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:08.484267203 +0000 UTC m=+1049.106575510" watchObservedRunningTime="2026-11-03 22:41:08.485078642 +0000 UTC m=+1049.107386994" Nov 03 22:41:08.917400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:08.917364 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:12.718835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.718786 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2"] Nov 03 22:41:12.723000 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.722976 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.772581 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.772544 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:12.819145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.819107 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:41:12.819145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.819119 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:41:12.819362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.819119 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:41:12.823059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.823029 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2"] Nov 03 22:41:12.826751 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.826720 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.826909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.826769 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.826967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.826942 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkqj\" (UniqueName: \"kubernetes.io/projected/5406d040-614e-4940-89e4-1220b6dcf2dc-kube-api-access-8jkqj\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.928509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.928472 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.928716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.928521 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.928716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.928628 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkqj\" (UniqueName: \"kubernetes.io/projected/5406d040-614e-4940-89e4-1220b6dcf2dc-kube-api-access-8jkqj\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.929224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.929195 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.931015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.930991 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:12.985942 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:12.985869 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkqj\" (UniqueName: \"kubernetes.io/projected/5406d040-614e-4940-89e4-1220b6dcf2dc-kube-api-access-8jkqj\") pod \"maas-api-revoke-sub-8d4f98ab-4zjv2\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:13.042958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:13.042922 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:13.418480 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:13.418440 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2"] Nov 03 22:41:13.423822 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:13.423780 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5406d040_614e_4940_89e4_1220b6dcf2dc.slice/crio-94612f98132a4972ed137a3c12983d4ca58f760ccf179fb4a299bac58d7d571a WatchSource:0}: Error finding container 94612f98132a4972ed137a3c12983d4ca58f760ccf179fb4a299bac58d7d571a: Status 404 returned error can't find the container with id 94612f98132a4972ed137a3c12983d4ca58f760ccf179fb4a299bac58d7d571a Nov 03 22:41:14.424424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:14.424392 5272 generic.go:356] "Generic (PLEG): container finished" podID="5406d040-614e-4940-89e4-1220b6dcf2dc" containerID="e0ea5742010ab75b9ab261908a4175a64020111acda5ec5bc99e1b49b736bc5a" exitCode=0 Nov 03 22:41:14.424870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:14.424474 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" event={"ID":"5406d040-614e-4940-89e4-1220b6dcf2dc","Type":"ContainerDied","Data":"e0ea5742010ab75b9ab261908a4175a64020111acda5ec5bc99e1b49b736bc5a"} Nov 03 22:41:14.424870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:14.424522 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" event={"ID":"5406d040-614e-4940-89e4-1220b6dcf2dc","Type":"ContainerStarted","Data":"94612f98132a4972ed137a3c12983d4ca58f760ccf179fb4a299bac58d7d571a"} Nov 03 22:41:15.576247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.576219 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:15.662923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.662888 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-openshift-ca\") pod \"5406d040-614e-4940-89e4-1220b6dcf2dc\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " Nov 03 22:41:15.663097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.662938 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5406d040-614e-4940-89e4-1220b6dcf2dc-kube-api-access-8jkqj\" (UniqueName: \"kubernetes.io/projected/5406d040-614e-4940-89e4-1220b6dcf2dc-kube-api-access-8jkqj\") pod \"5406d040-614e-4940-89e4-1220b6dcf2dc\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " Nov 03 22:41:15.663097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.663058 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-xks-ca\") pod \"5406d040-614e-4940-89e4-1220b6dcf2dc\" (UID: \"5406d040-614e-4940-89e4-1220b6dcf2dc\") " Nov 03 22:41:15.663252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.663229 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-openshift-ca" pod "5406d040-614e-4940-89e4-1220b6dcf2dc" (UID: "5406d040-614e-4940-89e4-1220b6dcf2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:15.663483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.663462 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:15.665043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.665019 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-xks-ca" pod "5406d040-614e-4940-89e4-1220b6dcf2dc" (UID: "5406d040-614e-4940-89e4-1220b6dcf2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:15.665043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.665024 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5406d040-614e-4940-89e4-1220b6dcf2dc-kube-api-access-8jkqj" pod "5406d040-614e-4940-89e4-1220b6dcf2dc" (UID: "5406d040-614e-4940-89e4-1220b6dcf2dc"). InnerVolumeSpecName "kube-api-access-8jkqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:15.764558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.764470 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkqj\" (UniqueName: \"kubernetes.io/projected/5406d040-614e-4940-89e4-1220b6dcf2dc-kube-api-access-8jkqj\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:15.764558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:15.764506 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5406d040-614e-4940-89e4-1220b6dcf2dc-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:16.440225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:16.440195 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" Nov 03 22:41:16.440225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:16.440211 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2" event={"ID":"5406d040-614e-4940-89e4-1220b6dcf2dc","Type":"ContainerDied","Data":"94612f98132a4972ed137a3c12983d4ca58f760ccf179fb4a299bac58d7d571a"} Nov 03 22:41:16.440496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:16.440266 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94612f98132a4972ed137a3c12983d4ca58f760ccf179fb4a299bac58d7d571a" Nov 03 22:41:16.918141 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:16.918092 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:16.931041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:16.931014 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:17.458894 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.458866 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:41:17.767752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.767659 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc"] Nov 03 22:41:17.768354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.768333 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5406d040-614e-4940-89e4-1220b6dcf2dc" containerName="revoke-keys" Nov 03 22:41:17.768436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.768357 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5406d040-614e-4940-89e4-1220b6dcf2dc" containerName="revoke-keys" Nov 03 22:41:17.768491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.768484 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5406d040-614e-4940-89e4-1220b6dcf2dc" containerName="revoke-keys" Nov 03 22:41:17.772128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.772107 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.821206 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.821174 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-961054-openshift-default-dockercfg-kjq9m\"" Nov 03 22:41:17.829831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.829774 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc"] Nov 03 22:41:17.909545 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909504 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f698-8f33-40d5-a91b-006f922f4427-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.909733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909563 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.909733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909663 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.909733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909704 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5gsj\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-kube-api-access-w5gsj\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.909955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909780 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.909955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909866 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.909955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909896 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f698-8f33-40d5-a91b-006f922f4427-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.910110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.909997 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:17.910110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:17.910033 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.011785 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.011741 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.011798 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.011959 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f698-8f33-40d5-a91b-006f922f4427-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012031 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012084 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012164 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5gsj\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-kube-api-access-w5gsj\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012173 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012200 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012262 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012285 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f698-8f33-40d5-a91b-006f922f4427-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012205 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012374 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012695 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.012999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.012972 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f698-8f33-40d5-a91b-006f922f4427-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.014885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.014859 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.015023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.015007 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f698-8f33-40d5-a91b-006f922f4427-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.069727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.069695 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.069859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.069796 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5gsj\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-kube-api-access-w5gsj\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.086443 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.086421 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:18.521586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.521493 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc"] Nov 03 22:41:18.526973 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:18.526942 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a53f698_8f33_40d5_a91b_006f922f4427.slice/crio-498ac1bab517894674d90487d0cbe255111fbeba203bf7960136a6e1e3515547 WatchSource:0}: Error finding container 498ac1bab517894674d90487d0cbe255111fbeba203bf7960136a6e1e3515547: Status 404 returned error can't find the container with id 498ac1bab517894674d90487d0cbe255111fbeba203bf7960136a6e1e3515547 Nov 03 22:41:18.529379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.529343 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:18.529491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.529425 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:18.529491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:18.529471 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:19.476755 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:19.476714 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" event={"ID":"9a53f698-8f33-40d5-a91b-006f922f4427","Type":"ContainerStarted","Data":"85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca"} Nov 03 22:41:19.477193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:19.476787 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" event={"ID":"9a53f698-8f33-40d5-a91b-006f922f4427","Type":"ContainerStarted","Data":"498ac1bab517894674d90487d0cbe255111fbeba203bf7960136a6e1e3515547"} Nov 03 22:41:19.642221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:19.642170 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podStartSLOduration=2.642155775 podStartE2EDuration="2.642155775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:19.638567335 +0000 UTC m=+1060.260875643" watchObservedRunningTime="2026-11-03 22:41:19.642155775 +0000 UTC m=+1060.264464083" Nov 03 22:41:20.087448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:20.087399 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:20.089386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:20.089358 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:41:20.089518 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:20.089417 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:41:21.087132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:21.087087 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:41:21.087770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:21.087735 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:41:22.087350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:22.087305 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:41:22.087776 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:22.087383 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:41:23.087709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:23.087670 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:41:23.088205 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:23.087729 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:41:24.020497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.020461 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp"] Nov 03 22:41:24.025213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.025187 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.087045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.087006 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:41:24.087214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.087082 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:41:24.100858 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.100826 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp"] Nov 03 22:41:24.111160 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.111128 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txzw\" (UniqueName: \"kubernetes.io/projected/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-kube-api-access-2txzw\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.111313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.111175 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-home\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.111313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.111248 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tls-certs\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.111442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.111336 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-model-cache\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.111442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.111409 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-dshm\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.111589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.111573 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tmp-dir\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.121624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.121604 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18c03a\"/\"distinct-2-w4-18c03a-kserve-self-signed-certs\"" Nov 03 22:41:24.212778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.212740 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tmp-dir\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.213555 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.213530 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txzw\" (UniqueName: \"kubernetes.io/projected/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-kube-api-access-2txzw\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.213742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.213728 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-home\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.213875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.213861 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tls-certs\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.217947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.217921 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-model-cache\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.218151 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.218135 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-dshm\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.229503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.218253 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-model-cache\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.229503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.217507 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-home\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.229503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.221974 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tls-certs\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.229503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.213401 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tmp-dir\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.232921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.232828 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-dshm\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.278969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.278869 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txzw\" (UniqueName: \"kubernetes.io/projected/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-kube-api-access-2txzw\") pod \"distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.337667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.337633 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:24.570527 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:41:24.570467 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:41:24.845634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:24.845608 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp"] Nov 03 22:41:24.847748 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:24.847721 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacae9a3d_2b1b_40e7_8a97_2a52d32a91da.slice/crio-4e485f473497142bd18b5144eb13e0bf5b91f53a742e1bd3df1d47a6ebd4a15a WatchSource:0}: Error finding container 4e485f473497142bd18b5144eb13e0bf5b91f53a742e1bd3df1d47a6ebd4a15a: Status 404 returned error can't find the container with id 4e485f473497142bd18b5144eb13e0bf5b91f53a742e1bd3df1d47a6ebd4a15a Nov 03 22:41:25.087833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:25.087771 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:41:25.088012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:25.087853 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:41:25.593460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:25.593423 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" event={"ID":"acae9a3d-2b1b-40e7-8a97-2a52d32a91da","Type":"ContainerStarted","Data":"432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f"} Nov 03 22:41:25.593909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:25.593476 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" event={"ID":"acae9a3d-2b1b-40e7-8a97-2a52d32a91da","Type":"ContainerStarted","Data":"4e485f473497142bd18b5144eb13e0bf5b91f53a742e1bd3df1d47a6ebd4a15a"} Nov 03 22:41:25.704098 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:25.704035 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podStartSLOduration=2.704019154 podStartE2EDuration="2.704019154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:25.703386064 +0000 UTC m=+1066.325694393" watchObservedRunningTime="2026-11-03 22:41:25.704019154 +0000 UTC m=+1066.326327461" Nov 03 22:41:26.091408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:26.091376 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:26.091858 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:26.091836 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:26.092502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:26.092484 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:41:26.338211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:26.338177 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:28.457576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.457537 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h"] Nov 03 22:41:28.461957 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.461933 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.495262 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.495231 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qthmb\" (UniqueName: \"kubernetes.io/projected/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-kube-api-access-qthmb\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.495410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.495292 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.495410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.495385 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.518383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.518350 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h"] Nov 03 22:41:28.560423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.560395 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:28.560423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.560404 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:41:28.560423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.560416 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:41:28.561745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.561724 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:41:28.597123 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.597089 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthmb\" (UniqueName: \"kubernetes.io/projected/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-kube-api-access-qthmb\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.597286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.597135 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.597286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.597195 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.597782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.597756 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.599542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.599518 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.654722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.654694 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthmb\" (UniqueName: \"kubernetes.io/projected/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-kube-api-access-qthmb\") pod \"maas-api-revoke-sub-3bc13563-bx77h\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:28.776676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:28.776596 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:29.151902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:29.151868 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h"] Nov 03 22:41:29.156723 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:29.156690 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f7dfee_3d60_4c21_9917_ee1f2e5ecce1.slice/crio-5f411c3ad53d2e029df5d5c4915627eb4a946407233b4cb00442c5ec448f718d WatchSource:0}: Error finding container 5f411c3ad53d2e029df5d5c4915627eb4a946407233b4cb00442c5ec448f718d: Status 404 returned error can't find the container with id 5f411c3ad53d2e029df5d5c4915627eb4a946407233b4cb00442c5ec448f718d Nov 03 22:41:29.626016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:29.625983 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" event={"ID":"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1","Type":"ContainerStarted","Data":"5f411c3ad53d2e029df5d5c4915627eb4a946407233b4cb00442c5ec448f718d"} Nov 03 22:41:30.633855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:30.633824 5272 generic.go:356] "Generic (PLEG): container finished" podID="55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" containerID="e9ce9402eb212f550972bbc4fca4291972ca715918da63a5e7889d2afb3dcc2b" exitCode=0 Nov 03 22:41:30.634274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:30.633885 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" event={"ID":"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1","Type":"ContainerDied","Data":"e9ce9402eb212f550972bbc4fca4291972ca715918da63a5e7889d2afb3dcc2b"} Nov 03 22:41:31.830609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.830582 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:31.939576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.939486 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-xks-ca\") pod \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " Nov 03 22:41:31.939576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.939539 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-openshift-ca\") pod \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " Nov 03 22:41:31.939764 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.939738 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-kube-api-access-qthmb\" (UniqueName: \"kubernetes.io/projected/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-kube-api-access-qthmb\") pod \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\" (UID: \"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1\") " Nov 03 22:41:31.939940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.939913 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-openshift-ca" pod "55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" (UID: "55f7dfee-3d60-4c21-9917-ee1f2e5ecce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:31.940274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.940252 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:31.941708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.941686 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-xks-ca" pod "55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" (UID: "55f7dfee-3d60-4c21-9917-ee1f2e5ecce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:31.941830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:31.941787 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-kube-api-access-qthmb" pod "55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" (UID: "55f7dfee-3d60-4c21-9917-ee1f2e5ecce1"). InnerVolumeSpecName "kube-api-access-qthmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:32.041760 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:32.041721 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qthmb\" (UniqueName: \"kubernetes.io/projected/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-kube-api-access-qthmb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:32.041760 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:32.041755 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:32.727004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:32.726968 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" Nov 03 22:41:32.727286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:32.726995 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h" event={"ID":"55f7dfee-3d60-4c21-9917-ee1f2e5ecce1","Type":"ContainerDied","Data":"5f411c3ad53d2e029df5d5c4915627eb4a946407233b4cb00442c5ec448f718d"} Nov 03 22:41:32.727372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:32.727314 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f411c3ad53d2e029df5d5c4915627eb4a946407233b4cb00442c5ec448f718d" Nov 03 22:41:33.051153 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:33.051107 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf"] Nov 03 22:41:33.054879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:33.054852 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2"] Nov 03 22:41:33.098328 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:33.098296 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fe534a-dcnn2"] Nov 03 22:41:33.101727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:33.101704 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f7d47e7-kpcnf"] Nov 03 22:41:33.500108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:33.500027 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eee0603-f52a-44ad-9616-016d8157330d" path="/var/lib/kubelet/pods/2eee0603-f52a-44ad-9616-016d8157330d/volumes" Nov 03 22:41:33.500504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:33.500490 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a" path="/var/lib/kubelet/pods/bdb179e2-f3a5-4540-8fc7-fb7eb55f8b6a/volumes" Nov 03 22:41:34.338232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:34.338195 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:34.351152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:34.351122 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:34.751587 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:34.751491 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:41:48.045915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:48.045876 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf"] Nov 03 22:41:48.092250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:48.092220 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ef24b-pd7mf"] Nov 03 22:41:49.368959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.368920 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf"] Nov 03 22:41:49.370656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.370627 5272 cpu_manager.go:414] "containerMap: removing container" podUID="55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" containerName="revoke-keys" Nov 03 22:41:49.370836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.370820 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" containerName="revoke-keys" Nov 03 22:41:49.371289 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.371269 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" containerName="revoke-keys" Nov 03 22:41:49.377230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.377205 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.475311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.475271 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.475634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.475608 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.475856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.475831 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.475966 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.475912 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.476023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.476000 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-kube-api-access-d2wpm\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.476097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.476030 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.476097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.476076 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.476189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.476098 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.476189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.476134 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.482319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.482296 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf"] Nov 03 22:41:49.495768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.495734 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa643b7-29d3-4eac-a45d-15bc0b90555a" path="/var/lib/kubelet/pods/4fa643b7-29d3-4eac-a45d-15bc0b90555a/volumes" Nov 03 22:41:49.577054 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577022 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577114 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-kube-api-access-d2wpm\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577146 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577188 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577210 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577244 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577352 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577386 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577438 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577485 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577498 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577719 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577828 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.577954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.577933 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.579523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.579496 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.579880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.579861 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.634905 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.634831 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.635039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.634999 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-kube-api-access-d2wpm\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:49.691004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:49.690970 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:50.128122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.128084 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf"] Nov 03 22:41:50.133236 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:50.133205 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5534eb_4e13_448a_a1a6_1628d4ff663c.slice/crio-58d063ba0b8c755c318544e617d98fab7326ce11e70ea4e7c603741e981b2be8 WatchSource:0}: Error finding container 58d063ba0b8c755c318544e617d98fab7326ce11e70ea4e7c603741e981b2be8: Status 404 returned error can't find the container with id 58d063ba0b8c755c318544e617d98fab7326ce11e70ea4e7c603741e981b2be8 Nov 03 22:41:50.135675 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.135639 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:50.135762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.135711 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:50.135762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.135742 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:50.880363 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.880319 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" event={"ID":"cc5534eb-4e13-448a-a1a6-1628d4ff663c","Type":"ContainerStarted","Data":"7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705"} Nov 03 22:41:50.880895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.880383 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" event={"ID":"cc5534eb-4e13-448a-a1a6-1628d4ff663c","Type":"ContainerStarted","Data":"58d063ba0b8c755c318544e617d98fab7326ce11e70ea4e7c603741e981b2be8"} Nov 03 22:41:50.924851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.924816 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:41:50.995922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:50.995869 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podStartSLOduration=1.995854062 podStartE2EDuration="1.995854062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:50.993237058 +0000 UTC m=+1091.615545366" watchObservedRunningTime="2026-11-03 22:41:50.995854062 +0000 UTC m=+1091.618162370" Nov 03 22:41:51.701569 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:51.701531 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:51.711967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:51.711909 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:41:51.712239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:51.712210 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:41:52.692828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:52.691629 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:41:52.692828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:52.691705 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:41:53.091913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.091863 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj"] Nov 03 22:41:53.098839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.094983 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c07e5fb8-flvhj"] Nov 03 22:41:53.529088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.528232 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df235505-0f1b-4b05-9d8a-957d00215ef8" path="/var/lib/kubelet/pods/df235505-0f1b-4b05-9d8a-957d00215ef8/volumes" Nov 03 22:41:53.697872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.695242 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:41:53.697872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.695312 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:41:53.886371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.886313 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj"] Nov 03 22:41:53.892964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.892941 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:53.977507 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.977298 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:53.977705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.977548 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwtn\" (UniqueName: \"kubernetes.io/projected/349bc540-633a-4a02-ad94-54d25cdbf610-kube-api-access-gwwtn\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:53.977705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.977635 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:53.985860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:53.985825 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj"] Nov 03 22:41:54.002103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.002057 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:41:54.002103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.002097 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:54.002352 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.002108 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:41:54.002352 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.002183 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:41:54.078906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.078865 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwtn\" (UniqueName: \"kubernetes.io/projected/349bc540-633a-4a02-ad94-54d25cdbf610-kube-api-access-gwwtn\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:54.079066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.078944 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:54.079066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.079032 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:54.079616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.079590 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:54.081510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.081488 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:54.139087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.139010 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwtn\" (UniqueName: \"kubernetes.io/projected/349bc540-633a-4a02-ad94-54d25cdbf610-kube-api-access-gwwtn\") pod \"maas-api-revoke-sub-d52e2b4a-x8pfj\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:54.207793 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.207755 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:54.503666 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:41:54.503588 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:41:54.614586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.614551 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj"] Nov 03 22:41:54.620402 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:54.620365 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349bc540_633a_4a02_ad94_54d25cdbf610.slice/crio-25671445fc5ae323de7ee8a33f20c34c3e0286ce44b77e74222efe2ef9e93ee9 WatchSource:0}: Error finding container 25671445fc5ae323de7ee8a33f20c34c3e0286ce44b77e74222efe2ef9e93ee9: Status 404 returned error can't find the container with id 25671445fc5ae323de7ee8a33f20c34c3e0286ce44b77e74222efe2ef9e93ee9 Nov 03 22:41:54.692239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.692203 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:41:54.692387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.692271 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:41:54.925005 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:54.924968 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" event={"ID":"349bc540-633a-4a02-ad94-54d25cdbf610","Type":"ContainerStarted","Data":"25671445fc5ae323de7ee8a33f20c34c3e0286ce44b77e74222efe2ef9e93ee9"} Nov 03 22:41:55.692121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:55.692084 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:41:55.692309 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:55.692156 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:41:55.933954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:55.933921 5272 generic.go:356] "Generic (PLEG): container finished" podID="349bc540-633a-4a02-ad94-54d25cdbf610" containerID="f6952af59f1ab021f29cdb091004a195b40d940bb7f20144ab5ec0a041d9c43a" exitCode=0 Nov 03 22:41:55.934343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:55.933978 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" event={"ID":"349bc540-633a-4a02-ad94-54d25cdbf610","Type":"ContainerDied","Data":"f6952af59f1ab021f29cdb091004a195b40d940bb7f20144ab5ec0a041d9c43a"} Nov 03 22:41:56.691886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:56.691849 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:41:56.692071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:56.691910 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:41:57.094362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.094332 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:57.219865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.216780 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349bc540-633a-4a02-ad94-54d25cdbf610-kube-api-access-gwwtn\" (UniqueName: \"kubernetes.io/projected/349bc540-633a-4a02-ad94-54d25cdbf610-kube-api-access-gwwtn\") pod \"349bc540-633a-4a02-ad94-54d25cdbf610\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " Nov 03 22:41:57.220226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.220200 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-xks-ca\") pod \"349bc540-633a-4a02-ad94-54d25cdbf610\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " Nov 03 22:41:57.220467 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.220447 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-openshift-ca\") pod \"349bc540-633a-4a02-ad94-54d25cdbf610\" (UID: \"349bc540-633a-4a02-ad94-54d25cdbf610\") " Nov 03 22:41:57.222425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.222398 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-openshift-ca" pod "349bc540-633a-4a02-ad94-54d25cdbf610" (UID: "349bc540-633a-4a02-ad94-54d25cdbf610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:57.244173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.244141 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349bc540-633a-4a02-ad94-54d25cdbf610-kube-api-access-gwwtn" pod "349bc540-633a-4a02-ad94-54d25cdbf610" (UID: "349bc540-633a-4a02-ad94-54d25cdbf610"). InnerVolumeSpecName "kube-api-access-gwwtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:57.284027 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.283985 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-xks-ca" pod "349bc540-633a-4a02-ad94-54d25cdbf610" (UID: "349bc540-633a-4a02-ad94-54d25cdbf610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:57.323585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.323544 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwwtn\" (UniqueName: \"kubernetes.io/projected/349bc540-633a-4a02-ad94-54d25cdbf610-kube-api-access-gwwtn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:57.323585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.323586 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:57.323875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.323605 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/349bc540-633a-4a02-ad94-54d25cdbf610-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:57.769900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.769855 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:57.771516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.771490 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:57.772355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.772333 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:41:57.949262 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.949235 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" Nov 03 22:41:57.949423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.949270 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj" event={"ID":"349bc540-633a-4a02-ad94-54d25cdbf610","Type":"ContainerDied","Data":"25671445fc5ae323de7ee8a33f20c34c3e0286ce44b77e74222efe2ef9e93ee9"} Nov 03 22:41:57.949423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:57.949324 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25671445fc5ae323de7ee8a33f20c34c3e0286ce44b77e74222efe2ef9e93ee9" Nov 03 22:41:58.719066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.719028 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj"] Nov 03 22:41:58.719737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.719717 5272 cpu_manager.go:414] "containerMap: removing container" podUID="349bc540-633a-4a02-ad94-54d25cdbf610" containerName="revoke-keys" Nov 03 22:41:58.719825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.719738 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349bc540-633a-4a02-ad94-54d25cdbf610" containerName="revoke-keys" Nov 03 22:41:58.719867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.719846 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="349bc540-633a-4a02-ad94-54d25cdbf610" containerName="revoke-keys" Nov 03 22:41:58.723063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.723043 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.772639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.772579 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:58.772639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.772631 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:41:58.772933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.772697 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:41:58.772933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.772752 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:41:58.777472 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.777445 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj"] Nov 03 22:41:58.841374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.841325 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.841558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.841435 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.841613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.841565 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/26337f4e-2fd4-405d-a69d-dabbd6824483-kube-api-access-ksvwh\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.942509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.942476 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/26337f4e-2fd4-405d-a69d-dabbd6824483-kube-api-access-ksvwh\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.942691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.942589 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.942691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.942622 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.943307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.943284 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:58.945062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:58.945041 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:59.000158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:59.000080 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/26337f4e-2fd4-405d-a69d-dabbd6824483-kube-api-access-ksvwh\") pod \"maas-api-revoke-sub-113434f6-mxsrj\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:59.037075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:59.037043 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:41:59.422261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:59.422212 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj"] Nov 03 22:41:59.427305 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:41:59.427277 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26337f4e_2fd4_405d_a69d_dabbd6824483.slice/crio-bff69007756a7a7e68793ce76121d6eae4b62797bc197b1296324e29fb4b191e WatchSource:0}: Error finding container bff69007756a7a7e68793ce76121d6eae4b62797bc197b1296324e29fb4b191e: Status 404 returned error can't find the container with id bff69007756a7a7e68793ce76121d6eae4b62797bc197b1296324e29fb4b191e Nov 03 22:41:59.966937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:59.966896 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" event={"ID":"26337f4e-2fd4-405d-a69d-dabbd6824483","Type":"ContainerStarted","Data":"68f4fc64f1f0db1de4b856c162fcbde55f4a92e3d4af2dba579728de852aac35"} Nov 03 22:41:59.967336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:41:59.966982 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" event={"ID":"26337f4e-2fd4-405d-a69d-dabbd6824483","Type":"ContainerStarted","Data":"bff69007756a7a7e68793ce76121d6eae4b62797bc197b1296324e29fb4b191e"} Nov 03 22:42:00.079594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:00.079529 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" podStartSLOduration=2.079512334 podStartE2EDuration="2.079512334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:00.075361947 +0000 UTC m=+1100.697670256" watchObservedRunningTime="2026-11-03 22:42:00.079512334 +0000 UTC m=+1100.701820643" Nov 03 22:42:00.976439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:00.976399 5272 generic.go:356] "Generic (PLEG): container finished" podID="26337f4e-2fd4-405d-a69d-dabbd6824483" containerID="68f4fc64f1f0db1de4b856c162fcbde55f4a92e3d4af2dba579728de852aac35" exitCode=0 Nov 03 22:42:00.976898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:00.976495 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" event={"ID":"26337f4e-2fd4-405d-a69d-dabbd6824483","Type":"ContainerDied","Data":"68f4fc64f1f0db1de4b856c162fcbde55f4a92e3d4af2dba579728de852aac35"} Nov 03 22:42:02.183955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.183929 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:42:02.285491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.285463 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-openshift-ca\") pod \"26337f4e-2fd4-405d-a69d-dabbd6824483\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " Nov 03 22:42:02.285693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.285684 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-xks-ca\") pod \"26337f4e-2fd4-405d-a69d-dabbd6824483\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " Nov 03 22:42:02.285766 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.285728 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26337f4e-2fd4-405d-a69d-dabbd6824483-kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/26337f4e-2fd4-405d-a69d-dabbd6824483-kube-api-access-ksvwh\") pod \"26337f4e-2fd4-405d-a69d-dabbd6824483\" (UID: \"26337f4e-2fd4-405d-a69d-dabbd6824483\") " Nov 03 22:42:02.285950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.285920 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-openshift-ca" pod "26337f4e-2fd4-405d-a69d-dabbd6824483" (UID: "26337f4e-2fd4-405d-a69d-dabbd6824483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:02.286179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.286162 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:02.287919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.287870 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-xks-ca" pod "26337f4e-2fd4-405d-a69d-dabbd6824483" (UID: "26337f4e-2fd4-405d-a69d-dabbd6824483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:02.287997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.287939 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26337f4e-2fd4-405d-a69d-dabbd6824483-kube-api-access-ksvwh" pod "26337f4e-2fd4-405d-a69d-dabbd6824483" (UID: "26337f4e-2fd4-405d-a69d-dabbd6824483"). InnerVolumeSpecName "kube-api-access-ksvwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:02.386962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.386917 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26337f4e-2fd4-405d-a69d-dabbd6824483-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:02.386962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.386957 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/26337f4e-2fd4-405d-a69d-dabbd6824483-kube-api-access-ksvwh\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:02.992696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.992655 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" event={"ID":"26337f4e-2fd4-405d-a69d-dabbd6824483","Type":"ContainerDied","Data":"bff69007756a7a7e68793ce76121d6eae4b62797bc197b1296324e29fb4b191e"} Nov 03 22:42:02.992696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.992700 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj" Nov 03 22:42:02.992945 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:02.992723 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff69007756a7a7e68793ce76121d6eae4b62797bc197b1296324e29fb4b191e" Nov 03 22:42:04.051749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.051712 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms"] Nov 03 22:42:04.098123 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.098091 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7390395-cjwms"] Nov 03 22:42:04.461625 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.461538 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq"] Nov 03 22:42:04.462173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.462158 5272 cpu_manager.go:414] "containerMap: removing container" podUID="26337f4e-2fd4-405d-a69d-dabbd6824483" containerName="revoke-keys" Nov 03 22:42:04.462242 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.462174 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26337f4e-2fd4-405d-a69d-dabbd6824483" containerName="revoke-keys" Nov 03 22:42:04.462303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.462295 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="26337f4e-2fd4-405d-a69d-dabbd6824483" containerName="revoke-keys" Nov 03 22:42:04.466093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.466077 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.551480 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.551447 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq"] Nov 03 22:42:04.615011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.614964 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a883e441-712e-4aa9-88a8-04e118a4b594-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615072 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d66r\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-kube-api-access-8d66r\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615120 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a883e441-712e-4aa9-88a8-04e118a4b594-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615163 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615187 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615242 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615342 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615453 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615435 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.615504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.615489 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717197 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717105 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a883e441-712e-4aa9-88a8-04e118a4b594-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717197 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717156 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717197 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717173 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717212 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717235 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717274 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717306 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717422 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a883e441-712e-4aa9-88a8-04e118a4b594-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717484 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d66r\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-kube-api-access-8d66r\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717611 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717659 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.717876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717743 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.718047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.717879 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.718191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.718169 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a883e441-712e-4aa9-88a8-04e118a4b594-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.719591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.719566 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.719835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.719791 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a883e441-712e-4aa9-88a8-04e118a4b594-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.775234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.775194 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.775389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.775251 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d66r\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-kube-api-access-8d66r\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:04.780136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:04.780118 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:05.211686 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:05.211647 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq"] Nov 03 22:42:05.218146 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:05.218118 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda883e441_712e_4aa9_88a8_04e118a4b594.slice/crio-6886d7623ad1120081e185f305e5fcfd82c0e10ae850a1bb7ee00c80577648c6 WatchSource:0}: Error finding container 6886d7623ad1120081e185f305e5fcfd82c0e10ae850a1bb7ee00c80577648c6: Status 404 returned error can't find the container with id 6886d7623ad1120081e185f305e5fcfd82c0e10ae850a1bb7ee00c80577648c6 Nov 03 22:42:05.220527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:05.220492 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:05.220606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:05.220561 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:05.220656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:05.220609 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:05.494990 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:05.494902 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358d085e-31a0-4a0a-9b80-fbbea541fed5" path="/var/lib/kubelet/pods/358d085e-31a0-4a0a-9b80-fbbea541fed5/volumes" Nov 03 22:42:06.017766 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:06.017728 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" event={"ID":"a883e441-712e-4aa9-88a8-04e118a4b594","Type":"ContainerStarted","Data":"25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e"} Nov 03 22:42:06.017971 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:06.017784 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" event={"ID":"a883e441-712e-4aa9-88a8-04e118a4b594","Type":"ContainerStarted","Data":"6886d7623ad1120081e185f305e5fcfd82c0e10ae850a1bb7ee00c80577648c6"} Nov 03 22:42:06.227815 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:06.227758 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podStartSLOduration=2.227743155 podStartE2EDuration="2.227743155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:06.178299573 +0000 UTC m=+1106.800607881" watchObservedRunningTime="2026-11-03 22:42:06.227743155 +0000 UTC m=+1106.850051463" Nov 03 22:42:06.781213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:06.781176 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:06.782949 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:06.782925 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:42:06.783067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:06.782973 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:42:07.781191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:07.781151 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:42:07.781599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:07.781216 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:42:08.781180 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:08.781145 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:42:08.781343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:08.781203 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:42:09.781394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:09.781360 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:42:09.781787 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:09.781419 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:42:10.780450 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:10.780410 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:42:10.780608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:10.780466 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:42:11.781043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:11.781007 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:42:11.781470 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:11.781071 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:42:12.726725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.726687 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt"] Nov 03 22:42:12.731077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.731053 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.780390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.780356 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18c03a-openshift-default-dockercfg-p87mq\"" Nov 03 22:42:12.785200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.785172 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:12.785641 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.785483 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:12.786288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.786267 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:42:12.831499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.831466 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt"] Nov 03 22:42:12.842710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.842670 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8g8\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-kube-api-access-qj8g8\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.842913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.842834 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.842913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.842875 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.843038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.843015 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.843183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.843063 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.843369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.843345 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.843437 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.843421 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.843577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.843558 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.843612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.843589 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944401 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944448 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944520 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944537 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944576 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8g8\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-kube-api-access-qj8g8\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944616 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944633 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944704 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.944947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944746 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.945111 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.944996 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.945111 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.945036 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.945111 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.945096 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.945276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.945238 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.945449 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.945427 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.946944 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.946922 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:12.947187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:12.947169 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:13.057341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.057310 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:13.058027 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.058004 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8g8\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-kube-api-access-qj8g8\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:13.346867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.346763 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:13.781049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.780948 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt"] Nov 03 22:42:13.822813 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:13.822769 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794b8bf8_0a32_4181_a5cc_0ffa219ae5d2.slice/crio-e9e28e8b39f95686dc6fa0500d29a83d226c341bf8686ccce6629b2bd7410b6c WatchSource:0}: Error finding container e9e28e8b39f95686dc6fa0500d29a83d226c341bf8686ccce6629b2bd7410b6c: Status 404 returned error can't find the container with id e9e28e8b39f95686dc6fa0500d29a83d226c341bf8686ccce6629b2bd7410b6c Nov 03 22:42:13.825288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.825255 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:13.825358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.825326 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:13.825400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.825368 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:13.957860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.957819 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn"] Nov 03 22:42:13.962665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:13.962642 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.012477 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.012450 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:14.012649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.012582 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:42:14.012702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.012657 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:42:14.012747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.012727 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:42:14.016662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.016635 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn"] Nov 03 22:42:14.060288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.060247 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.060498 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.060445 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.060621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.060593 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7rk\" (UniqueName: \"kubernetes.io/projected/f0ada9ed-67f8-44da-917e-4363da6de8d0-kube-api-access-8x7rk\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.124656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.124618 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" event={"ID":"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2","Type":"ContainerStarted","Data":"5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75"} Nov 03 22:42:14.124848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.124672 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" event={"ID":"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2","Type":"ContainerStarted","Data":"e9e28e8b39f95686dc6fa0500d29a83d226c341bf8686ccce6629b2bd7410b6c"} Nov 03 22:42:14.162374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.162315 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.162571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.162518 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.162701 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.162655 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7rk\" (UniqueName: \"kubernetes.io/projected/f0ada9ed-67f8-44da-917e-4363da6de8d0-kube-api-access-8x7rk\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.163200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.163156 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.165608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.165576 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.220778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.220747 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7rk\" (UniqueName: \"kubernetes.io/projected/f0ada9ed-67f8-44da-917e-4363da6de8d0-kube-api-access-8x7rk\") pod \"maas-api-revoke-sub-3889e75f-qlhzn\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.241526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.241469 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podStartSLOduration=2.241454772 podStartE2EDuration="2.241454772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:14.237459314 +0000 UTC m=+1114.859767621" watchObservedRunningTime="2026-11-03 22:42:14.241454772 +0000 UTC m=+1114.863763081" Nov 03 22:42:14.280396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.280355 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:14.349240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.347956 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:14.350668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.350325 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:42:14.350668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.350380 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:42:14.680880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:14.680750 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn"] Nov 03 22:42:14.686178 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:14.686141 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ada9ed_67f8_44da_917e_4363da6de8d0.slice/crio-2d8235322f8576704704918abc530890cfe30c601cef63e795610eeaff691ed9 WatchSource:0}: Error finding container 2d8235322f8576704704918abc530890cfe30c601cef63e795610eeaff691ed9: Status 404 returned error can't find the container with id 2d8235322f8576704704918abc530890cfe30c601cef63e795610eeaff691ed9 Nov 03 22:42:15.132868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:15.132825 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" event={"ID":"f0ada9ed-67f8-44da-917e-4363da6de8d0","Type":"ContainerStarted","Data":"9d136a0cff3aaf24a5dc984f52155c3217bd4d35124cbbe8ad200a54f581971d"} Nov 03 22:42:15.133281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:15.132887 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" event={"ID":"f0ada9ed-67f8-44da-917e-4363da6de8d0","Type":"ContainerStarted","Data":"2d8235322f8576704704918abc530890cfe30c601cef63e795610eeaff691ed9"} Nov 03 22:42:15.262753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:15.262674 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" podStartSLOduration=2.262649344 podStartE2EDuration="2.262649344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:15.242314965 +0000 UTC m=+1115.864623290" watchObservedRunningTime="2026-11-03 22:42:15.262649344 +0000 UTC m=+1115.884957655" Nov 03 22:42:15.358883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:15.351201 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:42:15.358883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:15.351267 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:42:16.172605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:16.172517 5272 generic.go:356] "Generic (PLEG): container finished" podID="f0ada9ed-67f8-44da-917e-4363da6de8d0" containerID="9d136a0cff3aaf24a5dc984f52155c3217bd4d35124cbbe8ad200a54f581971d" exitCode=0 Nov 03 22:42:16.173058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:16.172600 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" event={"ID":"f0ada9ed-67f8-44da-917e-4363da6de8d0","Type":"ContainerDied","Data":"9d136a0cff3aaf24a5dc984f52155c3217bd4d35124cbbe8ad200a54f581971d"} Nov 03 22:42:16.353831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:16.350176 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:42:16.353831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:16.350238 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:42:17.371573 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.371512 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:42:17.372136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.371598 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:42:17.477502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.477433 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:17.517388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.517354 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ada9ed-67f8-44da-917e-4363da6de8d0-kube-api-access-8x7rk\" (UniqueName: \"kubernetes.io/projected/f0ada9ed-67f8-44da-917e-4363da6de8d0-kube-api-access-8x7rk\") pod \"f0ada9ed-67f8-44da-917e-4363da6de8d0\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " Nov 03 22:42:17.517542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.517394 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-openshift-ca\") pod \"f0ada9ed-67f8-44da-917e-4363da6de8d0\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " Nov 03 22:42:17.517591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.517576 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-xks-ca\") pod \"f0ada9ed-67f8-44da-917e-4363da6de8d0\" (UID: \"f0ada9ed-67f8-44da-917e-4363da6de8d0\") " Nov 03 22:42:17.517854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.517797 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-openshift-ca" pod "f0ada9ed-67f8-44da-917e-4363da6de8d0" (UID: "f0ada9ed-67f8-44da-917e-4363da6de8d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:17.518136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.518122 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.519399 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.519374 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ada9ed-67f8-44da-917e-4363da6de8d0-kube-api-access-8x7rk" pod "f0ada9ed-67f8-44da-917e-4363da6de8d0" (UID: "f0ada9ed-67f8-44da-917e-4363da6de8d0"). InnerVolumeSpecName "kube-api-access-8x7rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.519555 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.519534 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-xks-ca" pod "f0ada9ed-67f8-44da-917e-4363da6de8d0" (UID: "f0ada9ed-67f8-44da-917e-4363da6de8d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:17.619313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.619273 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7rk\" (UniqueName: \"kubernetes.io/projected/f0ada9ed-67f8-44da-917e-4363da6de8d0-kube-api-access-8x7rk\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.619313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:17.619314 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ada9ed-67f8-44da-917e-4363da6de8d0-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.222119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.222092 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" Nov 03 22:42:18.222295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.222126 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn" event={"ID":"f0ada9ed-67f8-44da-917e-4363da6de8d0","Type":"ContainerDied","Data":"2d8235322f8576704704918abc530890cfe30c601cef63e795610eeaff691ed9"} Nov 03 22:42:18.222295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.222182 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8235322f8576704704918abc530890cfe30c601cef63e795610eeaff691ed9" Nov 03 22:42:18.336527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.336492 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9"] Nov 03 22:42:18.337100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.337084 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f0ada9ed-67f8-44da-917e-4363da6de8d0" containerName="revoke-keys" Nov 03 22:42:18.337165 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.337102 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ada9ed-67f8-44da-917e-4363da6de8d0" containerName="revoke-keys" Nov 03 22:42:18.337207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.337203 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ada9ed-67f8-44da-917e-4363da6de8d0" containerName="revoke-keys" Nov 03 22:42:18.340872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.340853 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.357833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.355891 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:42:18.357833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.355958 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:42:18.429035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.428995 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429085 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429112 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429210 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429298 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429333 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429386 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429413 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgnd\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-kube-api-access-8cgnd\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.429642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.429469 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.432481 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.432453 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9"] Nov 03 22:42:18.530553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.530471 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.530553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.530509 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.530851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.530703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.530851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.530822 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.530887 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.530992 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531032 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgnd\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-kube-api-access-8cgnd\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531144 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531229 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531232 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531475 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531502 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531671 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.531838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.531821 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.533302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.533273 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.533429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.533405 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.591827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.591775 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.591984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.591781 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgnd\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-kube-api-access-8cgnd\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:18.684448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:18.684411 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:19.135204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.135164 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9"] Nov 03 22:42:19.140231 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:19.140200 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01a4b7c_5d26_4dd8_847d_17b838f3c07d.slice/crio-6f0307aa727c3446e412897fe64dc3b81a4ead2bd2514983b91ee57241516479 WatchSource:0}: Error finding container 6f0307aa727c3446e412897fe64dc3b81a4ead2bd2514983b91ee57241516479: Status 404 returned error can't find the container with id 6f0307aa727c3446e412897fe64dc3b81a4ead2bd2514983b91ee57241516479 Nov 03 22:42:19.142688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.142654 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.142761 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.142728 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.142841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.142761 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:19.219825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.219740 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6"] Nov 03 22:42:19.225748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.225721 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.234699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.234669 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" event={"ID":"e01a4b7c-5d26-4dd8-847d-17b838f3c07d","Type":"ContainerStarted","Data":"6f0307aa727c3446e412897fe64dc3b81a4ead2bd2514983b91ee57241516479"} Nov 03 22:42:19.287440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.287027 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6"] Nov 03 22:42:19.377827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.374293 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:42:19.377827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.374351 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:42:19.383021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.382930 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.383380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.383359 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37834fae-53d5-4f2d-b1ef-c59ea9593019-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.383640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.383586 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.383828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.383792 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbt4z\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-kube-api-access-vbt4z\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.383953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.383940 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.384164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.384150 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.384264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.384251 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.384390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.384377 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.384497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.384484 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.486655 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.486708 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.486753 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.486790 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.486922 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.486950 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37834fae-53d5-4f2d-b1ef-c59ea9593019-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.487030 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.487057 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbt4z\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-kube-api-access-vbt4z\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.487088 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.487257 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.487391 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.487734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.487637 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.488865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.487854 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37834fae-53d5-4f2d-b1ef-c59ea9593019-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.488865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.488135 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.489874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.489852 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.491959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.491939 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.569839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.569759 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbt4z\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-kube-api-access-vbt4z\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.571239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.570287 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.764883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.761629 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv"] Nov 03 22:42:19.768830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.768331 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:19.807731 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.807693 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv"] Nov 03 22:42:19.818380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.818350 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:42:19.818564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.818481 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:42:19.818564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.818535 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:19.818663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.818590 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:42:19.852119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.852071 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:19.896713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.896675 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:19.896990 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.896775 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:19.896990 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.896832 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9hpm\" (UniqueName: \"kubernetes.io/projected/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-kube-api-access-c9hpm\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:19.998558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.998518 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:19.998737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.998577 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:19.998737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.998598 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9hpm\" (UniqueName: \"kubernetes.io/projected/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-kube-api-access-c9hpm\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:19.999313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:19.999285 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:20.001430 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.001396 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:20.056272 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.056238 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9hpm\" (UniqueName: \"kubernetes.io/projected/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-kube-api-access-c9hpm\") pod \"maas-api-revoke-sub-566a5589-k85kv\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:20.092865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.092829 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:20.291560 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.286567 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" event={"ID":"e01a4b7c-5d26-4dd8-847d-17b838f3c07d","Type":"ContainerStarted","Data":"28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f"} Nov 03 22:42:20.306194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.305226 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6"] Nov 03 22:42:20.311226 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:20.311199 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37834fae_53d5_4f2d_b1ef_c59ea9593019.slice/crio-bfed534402c9cdeb2102c43bb5b9c9bc38785d00a8f6fd937ea71e5d580e08d9 WatchSource:0}: Error finding container bfed534402c9cdeb2102c43bb5b9c9bc38785d00a8f6fd937ea71e5d580e08d9: Status 404 returned error can't find the container with id bfed534402c9cdeb2102c43bb5b9c9bc38785d00a8f6fd937ea71e5d580e08d9 Nov 03 22:42:20.313750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.313711 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:20.313885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.313836 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:20.313961 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.313887 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:20.370249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.366919 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.370249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.366981 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 22:42:20.403166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.403112 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podStartSLOduration=2.40309676 podStartE2EDuration="2.40309676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:20.400576789 +0000 UTC m=+1121.022885109" watchObservedRunningTime="2026-11-03 22:42:20.40309676 +0000 UTC m=+1121.025405068" Nov 03 22:42:20.511024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.510853 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv"] Nov 03 22:42:20.519609 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:20.519567 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0f39f4_0c5f_4b5a_8dd6_51a84c617a9a.slice/crio-0a9eae21acb13daa6d830c324b10d979382ba886a7dca7618f6d3f467e0ee322 WatchSource:0}: Error finding container 0a9eae21acb13daa6d830c324b10d979382ba886a7dca7618f6d3f467e0ee322: Status 404 returned error can't find the container with id 0a9eae21acb13daa6d830c324b10d979382ba886a7dca7618f6d3f467e0ee322 Nov 03 22:42:20.685414 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.685380 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:20.687428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.687394 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:42:20.687585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:20.687456 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:42:21.079936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.079877 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb"] Nov 03 22:42:21.081411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.081378 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb365bd-7kqxb"] Nov 03 22:42:21.306852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.297044 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" event={"ID":"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a","Type":"ContainerStarted","Data":"6d2c9593e617014ed2837598ddf6101e6f20c7e5bfe47411494ccf10800a9406"} Nov 03 22:42:21.306852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.297112 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" event={"ID":"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a","Type":"ContainerStarted","Data":"0a9eae21acb13daa6d830c324b10d979382ba886a7dca7618f6d3f467e0ee322"} Nov 03 22:42:21.306852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.299089 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" event={"ID":"37834fae-53d5-4f2d-b1ef-c59ea9593019","Type":"ContainerStarted","Data":"d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068"} Nov 03 22:42:21.306852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.299157 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" event={"ID":"37834fae-53d5-4f2d-b1ef-c59ea9593019","Type":"ContainerStarted","Data":"bfed534402c9cdeb2102c43bb5b9c9bc38785d00a8f6fd937ea71e5d580e08d9"} Nov 03 22:42:21.434551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.421621 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" podStartSLOduration=2.421598999 podStartE2EDuration="2.421598999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:21.421025241 +0000 UTC m=+1122.043333563" watchObservedRunningTime="2026-11-03 22:42:21.421598999 +0000 UTC m=+1122.043907309" Nov 03 22:42:21.510779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.510745 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7755c6-a357-4cf8-b40c-70057cceaa34" path="/var/lib/kubelet/pods/ed7755c6-a357-4cf8-b40c-70057cceaa34/volumes" Nov 03 22:42:21.511136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.511119 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:21.511454 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.511402 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:21.512772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.512751 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:42:21.536627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.536576 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podStartSLOduration=2.536562765 podStartE2EDuration="2.536562765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:21.534487411 +0000 UTC m=+1122.156795716" watchObservedRunningTime="2026-11-03 22:42:21.536562765 +0000 UTC m=+1122.158871073" Nov 03 22:42:21.688831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.687937 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.688831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.687997 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:42:21.870480 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.870341 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:21.871280 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.871251 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:42:21.876945 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:21.876899 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 22:42:22.312873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:22.312109 5272 generic.go:356] "Generic (PLEG): container finished" podID="3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" containerID="6d2c9593e617014ed2837598ddf6101e6f20c7e5bfe47411494ccf10800a9406" exitCode=0 Nov 03 22:42:22.312873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:22.312136 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" event={"ID":"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a","Type":"ContainerDied","Data":"6d2c9593e617014ed2837598ddf6101e6f20c7e5bfe47411494ccf10800a9406"} Nov 03 22:42:22.688890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:22.688281 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.688890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:22.688401 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:42:22.859860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:22.857251 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:42:22.859860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:22.857316 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 22:42:23.612880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.610598 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:23.692831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.692221 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.692831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.692287 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:42:23.704225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.704196 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-xks-ca\") pod \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " Nov 03 22:42:23.704586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.704570 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-kube-api-access-c9hpm\" (UniqueName: \"kubernetes.io/projected/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-kube-api-access-c9hpm\") pod \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " Nov 03 22:42:23.704780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.704764 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-openshift-ca\") pod \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\" (UID: \"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a\") " Nov 03 22:42:23.705061 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.705041 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-openshift-ca" pod "3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" (UID: "3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:23.705440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.705423 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:23.706162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.706144 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-xks-ca" pod "3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" (UID: "3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:23.708088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.708067 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-kube-api-access-c9hpm" pod "3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" (UID: "3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a"). InnerVolumeSpecName "kube-api-access-c9hpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:23.806774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.806741 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:23.806974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.806846 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9hpm\" (UniqueName: \"kubernetes.io/projected/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a-kube-api-access-c9hpm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:23.877882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.877127 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:42:23.877882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:23.877208 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 22:42:24.328500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:24.328462 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" Nov 03 22:42:24.328500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:24.328496 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv" event={"ID":"3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a","Type":"ContainerDied","Data":"0a9eae21acb13daa6d830c324b10d979382ba886a7dca7618f6d3f467e0ee322"} Nov 03 22:42:24.328732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:24.328546 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9eae21acb13daa6d830c324b10d979382ba886a7dca7618f6d3f467e0ee322" Nov 03 22:42:24.523832 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:42:24.523769 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:42:24.687886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:24.687196 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.687886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:24.687261 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:42:24.865829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:24.865247 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:42:24.865829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:24.865317 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 22:42:25.685379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:25.685335 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:42:25.685700 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:25.685407 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 22:42:25.853288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:25.853247 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:42:25.853571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:25.853543 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 22:42:26.854309 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:26.854264 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:42:26.855642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:26.855608 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 22:42:27.146976 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.146882 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:27.147387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.147370 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:27.148193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.148172 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:42:27.853785 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.853748 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d"] Nov 03 22:42:27.854760 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.854731 5272 cpu_manager.go:414] "containerMap: removing container" podUID="3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" containerName="revoke-keys" Nov 03 22:42:27.854760 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.854759 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" containerName="revoke-keys" Nov 03 22:42:27.855298 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.854990 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" containerName="revoke-keys" Nov 03 22:42:27.859323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.859292 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.916180 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.916137 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d"] Nov 03 22:42:27.970966 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.970926 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971211 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971270 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971360 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971481 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971424 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971539 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5b706b-d8dc-4e29-a625-65739631a25c-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971633 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971566 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971633 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971600 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6vm\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-kube-api-access-fr6vm\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:27.971743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:27.971633 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073066 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5b706b-d8dc-4e29-a625-65739631a25c-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073126 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073189 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6vm\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-kube-api-access-fr6vm\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073245 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073422 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073454 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073495 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073552 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073614 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.073706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073626 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.074022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.073992 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5b706b-d8dc-4e29-a625-65739631a25c-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.074300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.074116 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.074715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.074343 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.074870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.074566 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.078367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.078337 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.078779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.078737 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.131901 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.131785 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.132056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.131969 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6vm\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-kube-api-access-fr6vm\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.178585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.178543 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:28.354831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.349267 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:28.354831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.353814 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:28.362308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.361835 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 22:42:28.772353 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.772244 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d"] Nov 03 22:42:28.777724 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:28.777694 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5b706b_d8dc_4e29_a625_65739631a25c.slice/crio-5d77971fdb8cff16044587ca73fd18edff2d21faea22bcfe404848425e2b08ab WatchSource:0}: Error finding container 5d77971fdb8cff16044587ca73fd18edff2d21faea22bcfe404848425e2b08ab: Status 404 returned error can't find the container with id 5d77971fdb8cff16044587ca73fd18edff2d21faea22bcfe404848425e2b08ab Nov 03 22:42:28.780183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.780145 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:28.780288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.780230 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:28.780288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:28.780280 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:29.383630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:29.383592 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" event={"ID":"2b5b706b-d8dc-4e29-a625-65739631a25c","Type":"ContainerStarted","Data":"81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c"} Nov 03 22:42:29.384189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:29.383658 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" event={"ID":"2b5b706b-d8dc-4e29-a625-65739631a25c","Type":"ContainerStarted","Data":"5d77971fdb8cff16044587ca73fd18edff2d21faea22bcfe404848425e2b08ab"} Nov 03 22:42:29.503176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:29.503108 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podStartSLOduration=2.503088327 podStartE2EDuration="2.503088327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:29.500521768 +0000 UTC m=+1130.122830069" watchObservedRunningTime="2026-11-03 22:42:29.503088327 +0000 UTC m=+1130.125396636" Nov 03 22:42:30.186267 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:30.181173 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:30.203845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:30.191020 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:30.203845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:30.191140 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:31.071797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:31.071736 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7"] Nov 03 22:42:31.125739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:31.125705 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc898fd-v5xf7"] Nov 03 22:42:31.185304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:31.185263 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:31.185631 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:31.185600 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:31.606828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:31.605631 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12796649-4c89-4ae8-9f46-9a7f8a88c76e" path="/var/lib/kubelet/pods/12796649-4c89-4ae8-9f46-9a7f8a88c76e/volumes" Nov 03 22:42:31.933891 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:31.933349 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:32.180826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.180227 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:32.180826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.180286 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:32.352122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.350924 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:32.398876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.398823 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:32.399639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.399595 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:42:32.438471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.438428 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:42:32.454735 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.454691 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 03 22:42:32.605033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.604914 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:32.620814 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.620754 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:32.748789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.748726 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:32.858032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.857927 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:32.972830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:32.972178 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:33.186885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.181277 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:33.186885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.181352 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:33.402104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.400816 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:33.402104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.401402 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:42:33.606826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.606179 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:33.613683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.607156 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:42:33.638599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.638092 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 22:42:33.638599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.638591 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:42:33.804828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.721423 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:42:33.804828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.791904 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 03 22:42:33.804828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.792479 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:42:33.841752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.841609 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:42:33.860266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.860174 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:42:33.976139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.976096 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:42:33.977143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.977119 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:42:33.998573 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:33.998547 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:42:34.008541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.008511 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:42:34.180016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.179913 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:34.180016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.179972 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:34.608525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.608492 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s"] Nov 03 22:42:34.617874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.617844 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.620378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.618770 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s"] Nov 03 22:42:34.671317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.670257 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:42:34.671317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.670444 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:42:34.671317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.670498 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:34.671317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.670587 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:42:34.704907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.704869 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.705178 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.705158 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjjjh\" (UniqueName: \"kubernetes.io/projected/3f93f005-86fd-4354-aca2-bc2a792a1556-kube-api-access-kjjjh\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.705274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.705256 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.811871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.808704 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.811871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.808874 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjjjh\" (UniqueName: \"kubernetes.io/projected/3f93f005-86fd-4354-aca2-bc2a792a1556-kube-api-access-kjjjh\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.811871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.809013 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.812191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.811884 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.815823 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.815595 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.874584 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.874507 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjjjh\" (UniqueName: \"kubernetes.io/projected/3f93f005-86fd-4354-aca2-bc2a792a1556-kube-api-access-kjjjh\") pod \"maas-api-revoke-sub-1402d092-vg92s\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:34.939038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:34.938996 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:35.191930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:35.181273 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:35.191930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:35.181347 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:35.401541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:35.401500 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s"] Nov 03 22:42:35.777701 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:35.777642 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" event={"ID":"3f93f005-86fd-4354-aca2-bc2a792a1556","Type":"ContainerStarted","Data":"0c36eedd3957e282c2cc25c25e9b3f07dfed1757cdedb83c61e8a41809a88717"} Nov 03 22:42:36.178921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:36.178884 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:36.179104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:36.178946 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:36.788037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:36.788000 5272 generic.go:356] "Generic (PLEG): container finished" podID="3f93f005-86fd-4354-aca2-bc2a792a1556" containerID="65d0fa0f73bff0797c941ac296532bd63c6261cf1b4b262e662cb0649f6ace46" exitCode=0 Nov 03 22:42:36.788690 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:36.788265 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" event={"ID":"3f93f005-86fd-4354-aca2-bc2a792a1556","Type":"ContainerDied","Data":"65d0fa0f73bff0797c941ac296532bd63c6261cf1b4b262e662cb0649f6ace46"} Nov 03 22:42:37.179471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:37.179434 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:42:37.179644 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:37.179500 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 22:42:38.004733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.004704 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:38.100761 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.100675 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f93f005-86fd-4354-aca2-bc2a792a1556-kube-api-access-kjjjh\" (UniqueName: \"kubernetes.io/projected/3f93f005-86fd-4354-aca2-bc2a792a1556-kube-api-access-kjjjh\") pod \"3f93f005-86fd-4354-aca2-bc2a792a1556\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " Nov 03 22:42:38.100961 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.100773 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-xks-ca\") pod \"3f93f005-86fd-4354-aca2-bc2a792a1556\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " Nov 03 22:42:38.100961 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.100899 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-openshift-ca\") pod \"3f93f005-86fd-4354-aca2-bc2a792a1556\" (UID: \"3f93f005-86fd-4354-aca2-bc2a792a1556\") " Nov 03 22:42:38.101276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.101249 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-openshift-ca" pod "3f93f005-86fd-4354-aca2-bc2a792a1556" (UID: "3f93f005-86fd-4354-aca2-bc2a792a1556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:38.101462 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.101439 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.102883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.102858 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-xks-ca" pod "3f93f005-86fd-4354-aca2-bc2a792a1556" (UID: "3f93f005-86fd-4354-aca2-bc2a792a1556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:38.102993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.102979 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f93f005-86fd-4354-aca2-bc2a792a1556-kube-api-access-kjjjh" pod "3f93f005-86fd-4354-aca2-bc2a792a1556" (UID: "3f93f005-86fd-4354-aca2-bc2a792a1556"). InnerVolumeSpecName "kube-api-access-kjjjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.191636 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.191605 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:38.192266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.192193 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:38.193063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.193040 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:42:38.203204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.203177 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjjjh\" (UniqueName: \"kubernetes.io/projected/3f93f005-86fd-4354-aca2-bc2a792a1556-kube-api-access-kjjjh\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.203300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.203207 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f93f005-86fd-4354-aca2-bc2a792a1556-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.731705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.731669 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv"] Nov 03 22:42:38.732552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.732533 5272 cpu_manager.go:414] "containerMap: removing container" podUID="3f93f005-86fd-4354-aca2-bc2a792a1556" containerName="revoke-keys" Nov 03 22:42:38.732628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.732554 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f93f005-86fd-4354-aca2-bc2a792a1556" containerName="revoke-keys" Nov 03 22:42:38.732749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.732738 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f93f005-86fd-4354-aca2-bc2a792a1556" containerName="revoke-keys" Nov 03 22:42:38.738015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.737986 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.788672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.788629 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv"] Nov 03 22:42:38.818923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.818887 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ldg\" (UniqueName: \"kubernetes.io/projected/5886d361-f2f5-4d70-997a-5c496c8f0e1b-kube-api-access-44ldg\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.820897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.820870 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.821390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.821368 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.826225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.826192 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" Nov 03 22:42:38.827924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.827897 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s" event={"ID":"3f93f005-86fd-4354-aca2-bc2a792a1556","Type":"ContainerDied","Data":"0c36eedd3957e282c2cc25c25e9b3f07dfed1757cdedb83c61e8a41809a88717"} Nov 03 22:42:38.828054 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.827953 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c36eedd3957e282c2cc25c25e9b3f07dfed1757cdedb83c61e8a41809a88717" Nov 03 22:42:38.936418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.936330 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.936418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.936418 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.936645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.936608 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ldg\" (UniqueName: \"kubernetes.io/projected/5886d361-f2f5-4d70-997a-5c496c8f0e1b-kube-api-access-44ldg\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.937780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.937749 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:38.940028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:38.940003 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:39.015876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.012756 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ldg\" (UniqueName: \"kubernetes.io/projected/5886d361-f2f5-4d70-997a-5c496c8f0e1b-kube-api-access-44ldg\") pod \"maas-api-revoke-sub-fae3e05d-9n2nv\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:39.048038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.047998 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5"] Nov 03 22:42:39.054848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.053763 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:39.054848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.054410 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3efff7f-86tf5"] Nov 03 22:42:39.431332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.431295 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86"] Nov 03 22:42:39.437385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.437355 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.441788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.441761 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv"] Nov 03 22:42:39.562919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.555626 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.562919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.555760 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdrh\" (UniqueName: \"kubernetes.io/projected/dfc84625-017e-4cdc-aec8-cf6ee28c811e-kube-api-access-zgdrh\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.562919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.555894 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.605872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.605775 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7da17ed-305f-4c3f-8041-5f96776bde36" path="/var/lib/kubelet/pods/e7da17ed-305f-4c3f-8041-5f96776bde36/volumes" Nov 03 22:42:39.606163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.606145 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86"] Nov 03 22:42:39.656939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.656906 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.657110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.657060 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdrh\" (UniqueName: \"kubernetes.io/projected/dfc84625-017e-4cdc-aec8-cf6ee28c811e-kube-api-access-zgdrh\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.657283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.657257 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.657733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.657710 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.659436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.659411 5272 scope.go:122] "RemoveContainer" containerID="43d1efb0b14680eb3f0a0632459cf156ea27ff87eca450c7a1ffd68d2a94d413" Nov 03 22:42:39.659603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.659584 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.669870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.669841 5272 scope.go:122] "RemoveContainer" containerID="6aa685ee446297b038e8a57854c596fd180e5ee53687412c32e1528a23e4985e" Nov 03 22:42:39.679162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.679143 5272 scope.go:122] "RemoveContainer" containerID="0b6c889ca529c4893c5c707cf2eb2219dd542d8a962148b6595c68c9a6c84e08" Nov 03 22:42:39.714649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.714625 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdrh\" (UniqueName: \"kubernetes.io/projected/dfc84625-017e-4cdc-aec8-cf6ee28c811e-kube-api-access-zgdrh\") pod \"maas-api-revoke-sub-3783bd33-95m86\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.823921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.823887 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:39.833777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:39.833750 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" event={"ID":"5886d361-f2f5-4d70-997a-5c496c8f0e1b","Type":"ContainerStarted","Data":"2cf5d8df62cd3982936d51b51cc0760bc0a66f557e9f0e5170864cf6ff848c50"} Nov 03 22:42:40.212448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:40.212411 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86"] Nov 03 22:42:40.217993 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:40.217961 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc84625_017e_4cdc_aec8_cf6ee28c811e.slice/crio-cad491a201266c6ff4a224c497bfb4694044058662d0f7b6923a4c30469b0fca WatchSource:0}: Error finding container cad491a201266c6ff4a224c497bfb4694044058662d0f7b6923a4c30469b0fca: Status 404 returned error can't find the container with id cad491a201266c6ff4a224c497bfb4694044058662d0f7b6923a4c30469b0fca Nov 03 22:42:40.842926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:40.842798 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" event={"ID":"dfc84625-017e-4cdc-aec8-cf6ee28c811e","Type":"ContainerStarted","Data":"8cb244e907d698cb1518f13d350bcf8a61c0bb74956f34facf1d1e19a1c8b702"} Nov 03 22:42:40.842926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:40.842901 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" event={"ID":"dfc84625-017e-4cdc-aec8-cf6ee28c811e","Type":"ContainerStarted","Data":"cad491a201266c6ff4a224c497bfb4694044058662d0f7b6923a4c30469b0fca"} Nov 03 22:42:40.845031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:40.845004 5272 generic.go:356] "Generic (PLEG): container finished" podID="5886d361-f2f5-4d70-997a-5c496c8f0e1b" containerID="0757f868ade9558db5485ecf9ab5a53da91acb38c456b0988bb4d764765e213c" exitCode=0 Nov 03 22:42:40.845150 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:40.845035 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" event={"ID":"5886d361-f2f5-4d70-997a-5c496c8f0e1b","Type":"ContainerDied","Data":"0757f868ade9558db5485ecf9ab5a53da91acb38c456b0988bb4d764765e213c"} Nov 03 22:42:40.997581 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:40.997524 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" podStartSLOduration=1.997509626 podStartE2EDuration="1.997509626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:40.950656125 +0000 UTC m=+1141.572964434" watchObservedRunningTime="2026-11-03 22:42:40.997509626 +0000 UTC m=+1141.619817954" Nov 03 22:42:41.048385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:41.048351 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk"] Nov 03 22:42:41.052361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:41.052330 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0623e684-7cvhk"] Nov 03 22:42:41.494736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:41.494657 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f642e942-a25e-48cd-b026-7db11eafc9d0" path="/var/lib/kubelet/pods/f642e942-a25e-48cd-b026-7db11eafc9d0/volumes" Nov 03 22:42:41.855156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:41.855120 5272 generic.go:356] "Generic (PLEG): container finished" podID="dfc84625-017e-4cdc-aec8-cf6ee28c811e" containerID="8cb244e907d698cb1518f13d350bcf8a61c0bb74956f34facf1d1e19a1c8b702" exitCode=0 Nov 03 22:42:41.855368 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:41.855213 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" event={"ID":"dfc84625-017e-4cdc-aec8-cf6ee28c811e","Type":"ContainerDied","Data":"8cb244e907d698cb1518f13d350bcf8a61c0bb74956f34facf1d1e19a1c8b702"} Nov 03 22:42:42.016526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.016497 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:42.049995 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.049948 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc"] Nov 03 22:42:42.053692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.053659 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa9cf14-bbgkc"] Nov 03 22:42:42.090166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.090135 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-xks-ca\") pod \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " Nov 03 22:42:42.090333 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.090260 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-openshift-ca\") pod \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " Nov 03 22:42:42.090333 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.090300 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5886d361-f2f5-4d70-997a-5c496c8f0e1b-kube-api-access-44ldg\" (UniqueName: \"kubernetes.io/projected/5886d361-f2f5-4d70-997a-5c496c8f0e1b-kube-api-access-44ldg\") pod \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\" (UID: \"5886d361-f2f5-4d70-997a-5c496c8f0e1b\") " Nov 03 22:42:42.090601 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.090578 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-openshift-ca" pod "5886d361-f2f5-4d70-997a-5c496c8f0e1b" (UID: "5886d361-f2f5-4d70-997a-5c496c8f0e1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:42.090773 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.090755 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:42.092391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.092369 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-xks-ca" pod "5886d361-f2f5-4d70-997a-5c496c8f0e1b" (UID: "5886d361-f2f5-4d70-997a-5c496c8f0e1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:42.092493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.092431 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5886d361-f2f5-4d70-997a-5c496c8f0e1b-kube-api-access-44ldg" pod "5886d361-f2f5-4d70-997a-5c496c8f0e1b" (UID: "5886d361-f2f5-4d70-997a-5c496c8f0e1b"). InnerVolumeSpecName "kube-api-access-44ldg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:42.192240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.192150 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5886d361-f2f5-4d70-997a-5c496c8f0e1b-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:42.192240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.192186 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44ldg\" (UniqueName: \"kubernetes.io/projected/5886d361-f2f5-4d70-997a-5c496c8f0e1b-kube-api-access-44ldg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:42.864475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.864437 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" Nov 03 22:42:42.864475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.864464 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv" event={"ID":"5886d361-f2f5-4d70-997a-5c496c8f0e1b","Type":"ContainerDied","Data":"2cf5d8df62cd3982936d51b51cc0760bc0a66f557e9f0e5170864cf6ff848c50"} Nov 03 22:42:42.864988 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:42.864519 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf5d8df62cd3982936d51b51cc0760bc0a66f557e9f0e5170864cf6ff848c50" Nov 03 22:42:43.054527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.054497 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:43.205519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.205420 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-xks-ca\") pod \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " Nov 03 22:42:43.205519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.205495 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc84625-017e-4cdc-aec8-cf6ee28c811e-kube-api-access-zgdrh\" (UniqueName: \"kubernetes.io/projected/dfc84625-017e-4cdc-aec8-cf6ee28c811e-kube-api-access-zgdrh\") pod \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " Nov 03 22:42:43.205747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.205582 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-openshift-ca\") pod \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\" (UID: \"dfc84625-017e-4cdc-aec8-cf6ee28c811e\") " Nov 03 22:42:43.206909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.206856 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-openshift-ca" pod "dfc84625-017e-4cdc-aec8-cf6ee28c811e" (UID: "dfc84625-017e-4cdc-aec8-cf6ee28c811e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:43.208986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.208953 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-xks-ca" pod "dfc84625-017e-4cdc-aec8-cf6ee28c811e" (UID: "dfc84625-017e-4cdc-aec8-cf6ee28c811e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:43.208986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.208969 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc84625-017e-4cdc-aec8-cf6ee28c811e-kube-api-access-zgdrh" pod "dfc84625-017e-4cdc-aec8-cf6ee28c811e" (UID: "dfc84625-017e-4cdc-aec8-cf6ee28c811e"). InnerVolumeSpecName "kube-api-access-zgdrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:43.307432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.307393 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:43.307432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.307428 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdrh\" (UniqueName: \"kubernetes.io/projected/dfc84625-017e-4cdc-aec8-cf6ee28c811e-kube-api-access-zgdrh\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:43.307643 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.307445 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc84625-017e-4cdc-aec8-cf6ee28c811e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:43.494093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.494015 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed941de0-6465-4935-ac7a-6efa5bea567f" path="/var/lib/kubelet/pods/ed941de0-6465-4935-ac7a-6efa5bea567f/volumes" Nov 03 22:42:43.873432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.873397 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" Nov 03 22:42:43.873432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.873413 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86" event={"ID":"dfc84625-017e-4cdc-aec8-cf6ee28c811e","Type":"ContainerDied","Data":"cad491a201266c6ff4a224c497bfb4694044058662d0f7b6923a4c30469b0fca"} Nov 03 22:42:43.873879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:43.873471 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad491a201266c6ff4a224c497bfb4694044058662d0f7b6923a4c30469b0fca" Nov 03 22:42:44.822990 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.822943 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs"] Nov 03 22:42:44.823605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.823587 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5886d361-f2f5-4d70-997a-5c496c8f0e1b" containerName="revoke-keys" Nov 03 22:42:44.823680 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.823607 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5886d361-f2f5-4d70-997a-5c496c8f0e1b" containerName="revoke-keys" Nov 03 22:42:44.823680 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.823623 5272 cpu_manager.go:414] "containerMap: removing container" podUID="dfc84625-017e-4cdc-aec8-cf6ee28c811e" containerName="revoke-keys" Nov 03 22:42:44.823680 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.823628 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc84625-017e-4cdc-aec8-cf6ee28c811e" containerName="revoke-keys" Nov 03 22:42:44.823768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.823734 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5886d361-f2f5-4d70-997a-5c496c8f0e1b" containerName="revoke-keys" Nov 03 22:42:44.823768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.823744 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc84625-017e-4cdc-aec8-cf6ee28c811e" containerName="revoke-keys" Nov 03 22:42:44.828096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.828076 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:44.881622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.881586 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs"] Nov 03 22:42:44.922159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.922126 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:44.922378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.922134 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:42:44.922378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.922158 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:42:44.923471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.923453 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:42:44.930008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.929987 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:44.930088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.930063 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:44.930164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:44.930140 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lk8\" (UniqueName: \"kubernetes.io/projected/b3032e8a-f500-4fc8-90ae-b95156531a81-kube-api-access-67lk8\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.031789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.031754 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.032014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.031838 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lk8\" (UniqueName: \"kubernetes.io/projected/b3032e8a-f500-4fc8-90ae-b95156531a81-kube-api-access-67lk8\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.032014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.031986 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.032539 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.032514 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.034299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.034277 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.089495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.089426 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lk8\" (UniqueName: \"kubernetes.io/projected/b3032e8a-f500-4fc8-90ae-b95156531a81-kube-api-access-67lk8\") pod \"maas-api-revoke-sub-ec15edaa-ckmbs\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.141230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.141198 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:45.518402 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.518302 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs"] Nov 03 22:42:45.523902 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:45.523869 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3032e8a_f500_4fc8_90ae_b95156531a81.slice/crio-6652511010c5b9b629f187b1cc1df679f3d9268c52da7d36f408292dda56cc40 WatchSource:0}: Error finding container 6652511010c5b9b629f187b1cc1df679f3d9268c52da7d36f408292dda56cc40: Status 404 returned error can't find the container with id 6652511010c5b9b629f187b1cc1df679f3d9268c52da7d36f408292dda56cc40 Nov 03 22:42:45.890489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.890452 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" event={"ID":"b3032e8a-f500-4fc8-90ae-b95156531a81","Type":"ContainerStarted","Data":"c42add853035352c3270e1544ce5fc3aa26b1fb5801b53ac7278ca35f1bf94e0"} Nov 03 22:42:45.890926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:45.890547 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" event={"ID":"b3032e8a-f500-4fc8-90ae-b95156531a81","Type":"ContainerStarted","Data":"6652511010c5b9b629f187b1cc1df679f3d9268c52da7d36f408292dda56cc40"} Nov 03 22:42:46.002546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.002487 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" podStartSLOduration=2.00247302 podStartE2EDuration="2.00247302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:45.998726777 +0000 UTC m=+1146.621035085" watchObservedRunningTime="2026-11-03 22:42:46.00247302 +0000 UTC m=+1146.624781327" Nov 03 22:42:46.050254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.050217 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx"] Nov 03 22:42:46.054299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.054261 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e2ec86-9hbmx"] Nov 03 22:42:46.699892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.699854 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr"] Nov 03 22:42:46.704189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.704167 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.761094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.761018 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr"] Nov 03 22:42:46.858831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.858776 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.859010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.858909 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.859055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.859043 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v7p8\" (UniqueName: \"kubernetes.io/projected/513dbb9a-7533-4906-b95c-a12702e98ff1-kube-api-access-6v7p8\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.899074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.899040 5272 generic.go:356] "Generic (PLEG): container finished" podID="b3032e8a-f500-4fc8-90ae-b95156531a81" containerID="c42add853035352c3270e1544ce5fc3aa26b1fb5801b53ac7278ca35f1bf94e0" exitCode=0 Nov 03 22:42:46.899493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.899118 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" event={"ID":"b3032e8a-f500-4fc8-90ae-b95156531a81","Type":"ContainerDied","Data":"c42add853035352c3270e1544ce5fc3aa26b1fb5801b53ac7278ca35f1bf94e0"} Nov 03 22:42:46.960410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.960380 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.960609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.960432 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.960609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.960521 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v7p8\" (UniqueName: \"kubernetes.io/projected/513dbb9a-7533-4906-b95c-a12702e98ff1-kube-api-access-6v7p8\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.961147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.961122 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:46.962680 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:46.962653 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:47.019643 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:47.019566 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v7p8\" (UniqueName: \"kubernetes.io/projected/513dbb9a-7533-4906-b95c-a12702e98ff1-kube-api-access-6v7p8\") pod \"maas-api-revoke-sub-0cca8639-vl9pr\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:47.317388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:47.317355 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:47.494011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:47.493976 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d03a0a-a349-4700-b5a0-f0a286bd9469" path="/var/lib/kubelet/pods/16d03a0a-a349-4700-b5a0-f0a286bd9469/volumes" Nov 03 22:42:47.696233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:47.696147 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr"] Nov 03 22:42:47.701511 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:47.701480 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513dbb9a_7533_4906_b95c_a12702e98ff1.slice/crio-1d25e0ce80195a7a8d524054aa5c99e784613abd8fa5e717f1aab4a92d361024 WatchSource:0}: Error finding container 1d25e0ce80195a7a8d524054aa5c99e784613abd8fa5e717f1aab4a92d361024: Status 404 returned error can't find the container with id 1d25e0ce80195a7a8d524054aa5c99e784613abd8fa5e717f1aab4a92d361024 Nov 03 22:42:47.909403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:47.909359 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" event={"ID":"513dbb9a-7533-4906-b95c-a12702e98ff1","Type":"ContainerStarted","Data":"1d25e0ce80195a7a8d524054aa5c99e784613abd8fa5e717f1aab4a92d361024"} Nov 03 22:42:48.065438 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.065415 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:48.177105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.177070 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3032e8a-f500-4fc8-90ae-b95156531a81-kube-api-access-67lk8\" (UniqueName: \"kubernetes.io/projected/b3032e8a-f500-4fc8-90ae-b95156531a81-kube-api-access-67lk8\") pod \"b3032e8a-f500-4fc8-90ae-b95156531a81\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " Nov 03 22:42:48.177256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.177113 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-xks-ca\") pod \"b3032e8a-f500-4fc8-90ae-b95156531a81\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " Nov 03 22:42:48.177256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.177159 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-openshift-ca\") pod \"b3032e8a-f500-4fc8-90ae-b95156531a81\" (UID: \"b3032e8a-f500-4fc8-90ae-b95156531a81\") " Nov 03 22:42:48.177591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.177564 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-openshift-ca" pod "b3032e8a-f500-4fc8-90ae-b95156531a81" (UID: "b3032e8a-f500-4fc8-90ae-b95156531a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:48.177938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.177918 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:48.179280 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.179258 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-xks-ca" pod "b3032e8a-f500-4fc8-90ae-b95156531a81" (UID: "b3032e8a-f500-4fc8-90ae-b95156531a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:48.179357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.179336 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3032e8a-f500-4fc8-90ae-b95156531a81-kube-api-access-67lk8" pod "b3032e8a-f500-4fc8-90ae-b95156531a81" (UID: "b3032e8a-f500-4fc8-90ae-b95156531a81"). InnerVolumeSpecName "kube-api-access-67lk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:48.279561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.279473 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lk8\" (UniqueName: \"kubernetes.io/projected/b3032e8a-f500-4fc8-90ae-b95156531a81-kube-api-access-67lk8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:48.279561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.279512 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3032e8a-f500-4fc8-90ae-b95156531a81-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:48.918819 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.918778 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" Nov 03 22:42:48.919228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.918849 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs" event={"ID":"b3032e8a-f500-4fc8-90ae-b95156531a81","Type":"ContainerDied","Data":"6652511010c5b9b629f187b1cc1df679f3d9268c52da7d36f408292dda56cc40"} Nov 03 22:42:48.919228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.918931 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6652511010c5b9b629f187b1cc1df679f3d9268c52da7d36f408292dda56cc40" Nov 03 22:42:48.920774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.920746 5272 generic.go:356] "Generic (PLEG): container finished" podID="513dbb9a-7533-4906-b95c-a12702e98ff1" containerID="b6808fe39ecc8074dc0519866df7bf9d5542338c21505078a66aa33fe7aa5f90" exitCode=0 Nov 03 22:42:48.920915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:48.920850 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" event={"ID":"513dbb9a-7533-4906-b95c-a12702e98ff1","Type":"ContainerDied","Data":"b6808fe39ecc8074dc0519866df7bf9d5542338c21505078a66aa33fe7aa5f90"} Nov 03 22:42:49.083716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:49.083681 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r"] Nov 03 22:42:49.086927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:49.086898 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4ee703-wnj2r"] Nov 03 22:42:49.494476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:49.494395 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6902bdbb-4f72-4029-821c-20e92c574765" path="/var/lib/kubelet/pods/6902bdbb-4f72-4029-821c-20e92c574765/volumes" Nov 03 22:42:50.049057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.049023 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8"] Nov 03 22:42:50.052899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.052871 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09feca7a-pmlf8"] Nov 03 22:42:50.092740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.092712 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:50.209516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.209423 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-xks-ca\") pod \"513dbb9a-7533-4906-b95c-a12702e98ff1\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " Nov 03 22:42:50.209679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.209641 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513dbb9a-7533-4906-b95c-a12702e98ff1-kube-api-access-6v7p8\" (UniqueName: \"kubernetes.io/projected/513dbb9a-7533-4906-b95c-a12702e98ff1-kube-api-access-6v7p8\") pod \"513dbb9a-7533-4906-b95c-a12702e98ff1\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " Nov 03 22:42:50.209762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.209742 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-openshift-ca\") pod \"513dbb9a-7533-4906-b95c-a12702e98ff1\" (UID: \"513dbb9a-7533-4906-b95c-a12702e98ff1\") " Nov 03 22:42:50.210173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.210144 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-openshift-ca" pod "513dbb9a-7533-4906-b95c-a12702e98ff1" (UID: "513dbb9a-7533-4906-b95c-a12702e98ff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:50.210418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.210397 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:50.212229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.212205 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513dbb9a-7533-4906-b95c-a12702e98ff1-kube-api-access-6v7p8" pod "513dbb9a-7533-4906-b95c-a12702e98ff1" (UID: "513dbb9a-7533-4906-b95c-a12702e98ff1"). InnerVolumeSpecName "kube-api-access-6v7p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:50.212384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.212282 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-xks-ca" pod "513dbb9a-7533-4906-b95c-a12702e98ff1" (UID: "513dbb9a-7533-4906-b95c-a12702e98ff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:50.312225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.312187 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v7p8\" (UniqueName: \"kubernetes.io/projected/513dbb9a-7533-4906-b95c-a12702e98ff1-kube-api-access-6v7p8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:50.312225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.312218 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513dbb9a-7533-4906-b95c-a12702e98ff1-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:50.937537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.937509 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" Nov 03 22:42:50.937702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.937537 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr" event={"ID":"513dbb9a-7533-4906-b95c-a12702e98ff1","Type":"ContainerDied","Data":"1d25e0ce80195a7a8d524054aa5c99e784613abd8fa5e717f1aab4a92d361024"} Nov 03 22:42:50.937702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:50.937588 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d25e0ce80195a7a8d524054aa5c99e784613abd8fa5e717f1aab4a92d361024" Nov 03 22:42:51.493669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:51.493636 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a423bd9-60fc-4dd1-84d4-f10d5c845621" path="/var/lib/kubelet/pods/4a423bd9-60fc-4dd1-84d4-f10d5c845621/volumes" Nov 03 22:42:52.048678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.048640 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8"] Nov 03 22:42:52.095225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.095193 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c677a69-89hv8"] Nov 03 22:42:52.234085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.234050 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs"] Nov 03 22:42:52.234854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.234790 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b3032e8a-f500-4fc8-90ae-b95156531a81" containerName="revoke-keys" Nov 03 22:42:52.234854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.234824 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3032e8a-f500-4fc8-90ae-b95156531a81" containerName="revoke-keys" Nov 03 22:42:52.234854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.234858 5272 cpu_manager.go:414] "containerMap: removing container" podUID="513dbb9a-7533-4906-b95c-a12702e98ff1" containerName="revoke-keys" Nov 03 22:42:52.235086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.234867 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513dbb9a-7533-4906-b95c-a12702e98ff1" containerName="revoke-keys" Nov 03 22:42:52.235086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.235049 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="513dbb9a-7533-4906-b95c-a12702e98ff1" containerName="revoke-keys" Nov 03 22:42:52.235086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.235073 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3032e8a-f500-4fc8-90ae-b95156531a81" containerName="revoke-keys" Nov 03 22:42:52.239980 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.239960 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.289242 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.289209 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:52.289430 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.289285 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:42:52.289430 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.289325 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:42:52.289430 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.289402 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:42:52.293072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.293044 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs"] Nov 03 22:42:52.340227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.340144 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvxk\" (UniqueName: \"kubernetes.io/projected/7595d982-c998-4242-a084-3b3ad13b4c49-kube-api-access-scvxk\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.340367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.340277 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.340367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.340330 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.442172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.442134 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvxk\" (UniqueName: \"kubernetes.io/projected/7595d982-c998-4242-a084-3b3ad13b4c49-kube-api-access-scvxk\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.442385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.442230 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.442385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.442255 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.442944 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.442918 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.445050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.445028 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.499574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.499543 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvxk\" (UniqueName: \"kubernetes.io/projected/7595d982-c998-4242-a084-3b3ad13b4c49-kube-api-access-scvxk\") pod \"maas-api-revoke-sub-df037bec-hkgqs\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.553357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.553327 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:52.928753 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.928649 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs"] Nov 03 22:42:52.934191 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:52.934164 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7595d982_c998_4242_a084_3b3ad13b4c49.slice/crio-1d18fb94de840e86fd52a7665649f473dccd3c133a069f054b54b299db0ce13d WatchSource:0}: Error finding container 1d18fb94de840e86fd52a7665649f473dccd3c133a069f054b54b299db0ce13d: Status 404 returned error can't find the container with id 1d18fb94de840e86fd52a7665649f473dccd3c133a069f054b54b299db0ce13d Nov 03 22:42:52.954765 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:52.954737 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" event={"ID":"7595d982-c998-4242-a084-3b3ad13b4c49","Type":"ContainerStarted","Data":"1d18fb94de840e86fd52a7665649f473dccd3c133a069f054b54b299db0ce13d"} Nov 03 22:42:53.049847 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:53.049788 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9"] Nov 03 22:42:53.053580 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:53.053553 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95826b-xkjv9"] Nov 03 22:42:53.494292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:53.494257 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf234204-fcb1-4241-835b-f0a3d1e99b7b" path="/var/lib/kubelet/pods/bf234204-fcb1-4241-835b-f0a3d1e99b7b/volumes" Nov 03 22:42:53.494725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:53.494711 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47dae07-22e1-4ce8-a6ab-19750b713bb1" path="/var/lib/kubelet/pods/e47dae07-22e1-4ce8-a6ab-19750b713bb1/volumes" Nov 03 22:42:53.963713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:53.963659 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" event={"ID":"7595d982-c998-4242-a084-3b3ad13b4c49","Type":"ContainerStarted","Data":"753d6fd59dca02a82aa8b1f3e8323603ddbbd611609c4eac8282a8bae7514685"} Nov 03 22:42:54.078075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.078010 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" podStartSLOduration=2.077989577 podStartE2EDuration="2.077989577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:54.072272158 +0000 UTC m=+1154.694580467" watchObservedRunningTime="2026-11-03 22:42:54.077989577 +0000 UTC m=+1154.700297886" Nov 03 22:42:54.422494 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.422443 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5"] Nov 03 22:42:54.426708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.426692 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.481159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.481118 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5"] Nov 03 22:42:54.486919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.486890 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.487073 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.487022 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.487148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.487132 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqcc\" (UniqueName: \"kubernetes.io/projected/d4a427e0-1ec9-4326-98fd-643a25fd28ac-kube-api-access-rwqcc\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.509012 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:42:54.508972 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:42:54.588594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.588559 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.588766 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.588679 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.588823 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.588776 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqcc\" (UniqueName: \"kubernetes.io/projected/d4a427e0-1ec9-4326-98fd-643a25fd28ac-kube-api-access-rwqcc\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.589362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.589335 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.591261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.591233 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.646510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.646478 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqcc\" (UniqueName: \"kubernetes.io/projected/d4a427e0-1ec9-4326-98fd-643a25fd28ac-kube-api-access-rwqcc\") pod \"maas-api-revoke-sub-043c526f-n8mw5\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.740468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.740391 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:54.972779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.972734 5272 generic.go:356] "Generic (PLEG): container finished" podID="7595d982-c998-4242-a084-3b3ad13b4c49" containerID="753d6fd59dca02a82aa8b1f3e8323603ddbbd611609c4eac8282a8bae7514685" exitCode=0 Nov 03 22:42:54.973235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:54.972873 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" event={"ID":"7595d982-c998-4242-a084-3b3ad13b4c49","Type":"ContainerDied","Data":"753d6fd59dca02a82aa8b1f3e8323603ddbbd611609c4eac8282a8bae7514685"} Nov 03 22:42:55.125120 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:55.125086 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a427e0_1ec9_4326_98fd_643a25fd28ac.slice/crio-96feffd4564b9bafc6dbe5d432777167509b724a502c57c16a32f1459f5766af WatchSource:0}: Error finding container 96feffd4564b9bafc6dbe5d432777167509b724a502c57c16a32f1459f5766af: Status 404 returned error can't find the container with id 96feffd4564b9bafc6dbe5d432777167509b724a502c57c16a32f1459f5766af Nov 03 22:42:55.144671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:55.144643 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5"] Nov 03 22:42:55.971161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:55.971121 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m"] Nov 03 22:42:55.975038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:55.975010 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:55.983054 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:55.983023 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" event={"ID":"d4a427e0-1ec9-4326-98fd-643a25fd28ac","Type":"ContainerStarted","Data":"257c29a73804e02b30efa9e7862c6203f23eafe084fbbd96a5d9f55aa308445e"} Nov 03 22:42:55.983177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:55.983079 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" event={"ID":"d4a427e0-1ec9-4326-98fd-643a25fd28ac","Type":"ContainerStarted","Data":"96feffd4564b9bafc6dbe5d432777167509b724a502c57c16a32f1459f5766af"} Nov 03 22:42:56.041897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.041410 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m"] Nov 03 22:42:56.108905 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.108880 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.109064 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.109047 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.109176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.109162 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfqm\" (UniqueName: \"kubernetes.io/projected/6713c7e9-c02e-4833-a9a3-d3646d01ea71-kube-api-access-smfqm\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.133018 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.132994 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:56.192180 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.192126 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" podStartSLOduration=2.192111374 podStartE2EDuration="2.192111374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:56.189334709 +0000 UTC m=+1156.811643016" watchObservedRunningTime="2026-11-03 22:42:56.192111374 +0000 UTC m=+1156.814419682" Nov 03 22:42:56.210064 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210039 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-openshift-ca\") pod \"7595d982-c998-4242-a084-3b3ad13b4c49\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " Nov 03 22:42:56.210207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210143 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-xks-ca\") pod \"7595d982-c998-4242-a084-3b3ad13b4c49\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " Nov 03 22:42:56.210264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210208 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7595d982-c998-4242-a084-3b3ad13b4c49-kube-api-access-scvxk\" (UniqueName: \"kubernetes.io/projected/7595d982-c998-4242-a084-3b3ad13b4c49-kube-api-access-scvxk\") pod \"7595d982-c998-4242-a084-3b3ad13b4c49\" (UID: \"7595d982-c998-4242-a084-3b3ad13b4c49\") " Nov 03 22:42:56.210391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210369 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-openshift-ca" pod "7595d982-c998-4242-a084-3b3ad13b4c49" (UID: "7595d982-c998-4242-a084-3b3ad13b4c49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:56.210559 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210540 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.210645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210579 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfqm\" (UniqueName: \"kubernetes.io/projected/6713c7e9-c02e-4833-a9a3-d3646d01ea71-kube-api-access-smfqm\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.210774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210761 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.210985 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.210874 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:56.211510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.211484 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.212590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.212565 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7595d982-c998-4242-a084-3b3ad13b4c49-kube-api-access-scvxk" pod "7595d982-c998-4242-a084-3b3ad13b4c49" (UID: "7595d982-c998-4242-a084-3b3ad13b4c49"). InnerVolumeSpecName "kube-api-access-scvxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:56.212720 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.212701 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-xks-ca" pod "7595d982-c998-4242-a084-3b3ad13b4c49" (UID: "7595d982-c998-4242-a084-3b3ad13b4c49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:56.213065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.213049 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.267736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.267713 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfqm\" (UniqueName: \"kubernetes.io/projected/6713c7e9-c02e-4833-a9a3-d3646d01ea71-kube-api-access-smfqm\") pod \"maas-api-revoke-sub-673aded6-b2m2m\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.291141 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.291113 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:56.311791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.311764 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7595d982-c998-4242-a084-3b3ad13b4c49-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:56.311924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.311791 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvxk\" (UniqueName: \"kubernetes.io/projected/7595d982-c998-4242-a084-3b3ad13b4c49-kube-api-access-scvxk\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:56.680719 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.680685 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m"] Nov 03 22:42:56.685784 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:42:56.685755 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6713c7e9_c02e_4833_a9a3_d3646d01ea71.slice/crio-1b56aaa3589ed2b98ab1926c56fe0bef73a36744304ed2073ad61d6465b523e1 WatchSource:0}: Error finding container 1b56aaa3589ed2b98ab1926c56fe0bef73a36744304ed2073ad61d6465b523e1: Status 404 returned error can't find the container with id 1b56aaa3589ed2b98ab1926c56fe0bef73a36744304ed2073ad61d6465b523e1 Nov 03 22:42:56.992639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.992603 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" event={"ID":"6713c7e9-c02e-4833-a9a3-d3646d01ea71","Type":"ContainerStarted","Data":"bf6b0c2e3aad7b0eb3309795d94cfb3a269277d56e0fd450e9ed8813ca63da0b"} Nov 03 22:42:56.993196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.992697 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" event={"ID":"6713c7e9-c02e-4833-a9a3-d3646d01ea71","Type":"ContainerStarted","Data":"1b56aaa3589ed2b98ab1926c56fe0bef73a36744304ed2073ad61d6465b523e1"} Nov 03 22:42:56.994374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.994350 5272 generic.go:356] "Generic (PLEG): container finished" podID="d4a427e0-1ec9-4326-98fd-643a25fd28ac" containerID="257c29a73804e02b30efa9e7862c6203f23eafe084fbbd96a5d9f55aa308445e" exitCode=0 Nov 03 22:42:56.994486 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.994441 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" event={"ID":"d4a427e0-1ec9-4326-98fd-643a25fd28ac","Type":"ContainerDied","Data":"257c29a73804e02b30efa9e7862c6203f23eafe084fbbd96a5d9f55aa308445e"} Nov 03 22:42:56.995770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.995751 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" event={"ID":"7595d982-c998-4242-a084-3b3ad13b4c49","Type":"ContainerDied","Data":"1d18fb94de840e86fd52a7665649f473dccd3c133a069f054b54b299db0ce13d"} Nov 03 22:42:56.995884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.995785 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d18fb94de840e86fd52a7665649f473dccd3c133a069f054b54b299db0ce13d" Nov 03 22:42:56.995884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:56.995797 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs" Nov 03 22:42:57.100944 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:57.100894 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" podStartSLOduration=2.100877118 podStartE2EDuration="2.100877118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:57.100566027 +0000 UTC m=+1157.722874336" watchObservedRunningTime="2026-11-03 22:42:57.100877118 +0000 UTC m=+1157.723185428" Nov 03 22:42:58.006421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.006330 5272 generic.go:356] "Generic (PLEG): container finished" podID="6713c7e9-c02e-4833-a9a3-d3646d01ea71" containerID="bf6b0c2e3aad7b0eb3309795d94cfb3a269277d56e0fd450e9ed8813ca63da0b" exitCode=0 Nov 03 22:42:58.006421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.006408 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" event={"ID":"6713c7e9-c02e-4833-a9a3-d3646d01ea71","Type":"ContainerDied","Data":"bf6b0c2e3aad7b0eb3309795d94cfb3a269277d56e0fd450e9ed8813ca63da0b"} Nov 03 22:42:58.174172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.174146 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:58.234914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.234879 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-xks-ca\") pod \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " Nov 03 22:42:58.235107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.234976 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-openshift-ca\") pod \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " Nov 03 22:42:58.235194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.235118 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a427e0-1ec9-4326-98fd-643a25fd28ac-kube-api-access-rwqcc\" (UniqueName: \"kubernetes.io/projected/d4a427e0-1ec9-4326-98fd-643a25fd28ac-kube-api-access-rwqcc\") pod \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\" (UID: \"d4a427e0-1ec9-4326-98fd-643a25fd28ac\") " Nov 03 22:42:58.235374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.235349 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-openshift-ca" pod "d4a427e0-1ec9-4326-98fd-643a25fd28ac" (UID: "d4a427e0-1ec9-4326-98fd-643a25fd28ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:58.235733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.235711 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:58.237189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.237166 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a427e0-1ec9-4326-98fd-643a25fd28ac-kube-api-access-rwqcc" pod "d4a427e0-1ec9-4326-98fd-643a25fd28ac" (UID: "d4a427e0-1ec9-4326-98fd-643a25fd28ac"). InnerVolumeSpecName "kube-api-access-rwqcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:58.237269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.237219 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-xks-ca" pod "d4a427e0-1ec9-4326-98fd-643a25fd28ac" (UID: "d4a427e0-1ec9-4326-98fd-643a25fd28ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:58.337550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.337514 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a427e0-1ec9-4326-98fd-643a25fd28ac-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:58.337550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:58.337545 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqcc\" (UniqueName: \"kubernetes.io/projected/d4a427e0-1ec9-4326-98fd-643a25fd28ac-kube-api-access-rwqcc\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.059933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.059848 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" Nov 03 22:42:59.066619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.060231 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5" event={"ID":"d4a427e0-1ec9-4326-98fd-643a25fd28ac","Type":"ContainerDied","Data":"96feffd4564b9bafc6dbe5d432777167509b724a502c57c16a32f1459f5766af"} Nov 03 22:42:59.066619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.064377 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96feffd4564b9bafc6dbe5d432777167509b724a502c57c16a32f1459f5766af" Nov 03 22:42:59.253021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.252994 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:42:59.364090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.363994 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-xks-ca\") pod \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " Nov 03 22:42:59.364267 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.364089 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-openshift-ca\") pod \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " Nov 03 22:42:59.364267 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.364247 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6713c7e9-c02e-4833-a9a3-d3646d01ea71-kube-api-access-smfqm\" (UniqueName: \"kubernetes.io/projected/6713c7e9-c02e-4833-a9a3-d3646d01ea71-kube-api-access-smfqm\") pod \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\" (UID: \"6713c7e9-c02e-4833-a9a3-d3646d01ea71\") " Nov 03 22:42:59.364520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.364490 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-openshift-ca" pod "6713c7e9-c02e-4833-a9a3-d3646d01ea71" (UID: "6713c7e9-c02e-4833-a9a3-d3646d01ea71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:59.364953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.364928 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.366648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.366623 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-xks-ca" pod "6713c7e9-c02e-4833-a9a3-d3646d01ea71" (UID: "6713c7e9-c02e-4833-a9a3-d3646d01ea71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:59.366865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.366842 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6713c7e9-c02e-4833-a9a3-d3646d01ea71-kube-api-access-smfqm" pod "6713c7e9-c02e-4833-a9a3-d3646d01ea71" (UID: "6713c7e9-c02e-4833-a9a3-d3646d01ea71"). InnerVolumeSpecName "kube-api-access-smfqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:59.466161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.466127 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6713c7e9-c02e-4833-a9a3-d3646d01ea71-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:59.466161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:42:59.466155 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfqm\" (UniqueName: \"kubernetes.io/projected/6713c7e9-c02e-4833-a9a3-d3646d01ea71-kube-api-access-smfqm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:00.069465 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:00.069429 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" event={"ID":"6713c7e9-c02e-4833-a9a3-d3646d01ea71","Type":"ContainerDied","Data":"1b56aaa3589ed2b98ab1926c56fe0bef73a36744304ed2073ad61d6465b523e1"} Nov 03 22:43:00.069886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:00.069478 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m" Nov 03 22:43:00.069886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:00.069523 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b56aaa3589ed2b98ab1926c56fe0bef73a36744304ed2073ad61d6465b523e1" Nov 03 22:43:02.209936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.209658 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n"] Nov 03 22:43:02.210447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210427 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d4a427e0-1ec9-4326-98fd-643a25fd28ac" containerName="revoke-keys" Nov 03 22:43:02.210510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210447 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a427e0-1ec9-4326-98fd-643a25fd28ac" containerName="revoke-keys" Nov 03 22:43:02.210510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210462 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7595d982-c998-4242-a084-3b3ad13b4c49" containerName="revoke-keys" Nov 03 22:43:02.210510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210467 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7595d982-c998-4242-a084-3b3ad13b4c49" containerName="revoke-keys" Nov 03 22:43:02.210510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210476 5272 cpu_manager.go:414] "containerMap: removing container" podUID="6713c7e9-c02e-4833-a9a3-d3646d01ea71" containerName="revoke-keys" Nov 03 22:43:02.210510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210481 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6713c7e9-c02e-4833-a9a3-d3646d01ea71" containerName="revoke-keys" Nov 03 22:43:02.210743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210606 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7595d982-c998-4242-a084-3b3ad13b4c49" containerName="revoke-keys" Nov 03 22:43:02.210743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210615 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="6713c7e9-c02e-4833-a9a3-d3646d01ea71" containerName="revoke-keys" Nov 03 22:43:02.210743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.210622 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a427e0-1ec9-4326-98fd-643a25fd28ac" containerName="revoke-keys" Nov 03 22:43:02.214538 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.214515 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.268234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.268202 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n"] Nov 03 22:43:02.306774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.306739 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.306951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.306793 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.306951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.306937 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghk2r\" (UniqueName: \"kubernetes.io/projected/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-kube-api-access-ghk2r\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.309618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.309592 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:02.309618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.309592 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:43:02.309780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.309664 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:43:02.309780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.309671 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:43:02.407848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.407787 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.408024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.407875 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.408024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.408013 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghk2r\" (UniqueName: \"kubernetes.io/projected/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-kube-api-access-ghk2r\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.408578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.408550 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.410258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.410234 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.465238 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.465174 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghk2r\" (UniqueName: \"kubernetes.io/projected/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-kube-api-access-ghk2r\") pod \"maas-api-revoke-sub-8230bae9-td24n\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.529461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.529426 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:02.912485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.912446 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n"] Nov 03 22:43:02.918470 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:02.918372 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1a2bd2_a8da_439c_b11c_d9cf1cc52ffc.slice/crio-9688b1d602fd80fa020a503b402e20bfb130cea9d53a7c83f6a96ed5ebced458 WatchSource:0}: Error finding container 9688b1d602fd80fa020a503b402e20bfb130cea9d53a7c83f6a96ed5ebced458: Status 404 returned error can't find the container with id 9688b1d602fd80fa020a503b402e20bfb130cea9d53a7c83f6a96ed5ebced458 Nov 03 22:43:02.998281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:02.998236 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm"] Nov 03 22:43:03.003992 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.003967 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.050556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.050525 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk"] Nov 03 22:43:03.055012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.054984 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb384dd-l89pk"] Nov 03 22:43:03.058683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.058659 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm"] Nov 03 22:43:03.096102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.096071 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" event={"ID":"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc","Type":"ContainerStarted","Data":"9688b1d602fd80fa020a503b402e20bfb130cea9d53a7c83f6a96ed5ebced458"} Nov 03 22:43:03.118496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.118461 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.118496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.118497 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww92t\" (UniqueName: \"kubernetes.io/projected/8f99857d-6f80-421c-b70d-5efb80a73712-kube-api-access-ww92t\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.118690 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.118665 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.220185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.220087 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.220185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.220131 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww92t\" (UniqueName: \"kubernetes.io/projected/8f99857d-6f80-421c-b70d-5efb80a73712-kube-api-access-ww92t\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.220676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.220308 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.220940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.220918 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.222422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.222403 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.277875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.277845 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww92t\" (UniqueName: \"kubernetes.io/projected/8f99857d-6f80-421c-b70d-5efb80a73712-kube-api-access-ww92t\") pod \"maas-api-revoke-sub-7370ae43-dzlpm\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.320218 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.320185 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:03.493504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.493428 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a61f3ee-8181-4669-a191-0d6dba217449" path="/var/lib/kubelet/pods/5a61f3ee-8181-4669-a191-0d6dba217449/volumes" Nov 03 22:43:03.584628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.584599 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft"] Nov 03 22:43:03.589308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.589286 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.669036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.669002 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft"] Nov 03 22:43:03.702490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.702448 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm"] Nov 03 22:43:03.705374 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:03.705344 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f99857d_6f80_421c_b70d_5efb80a73712.slice/crio-079a99fde76d32d63207edf291e335096225d2c0fa026014eb1b6b02aa01e598 WatchSource:0}: Error finding container 079a99fde76d32d63207edf291e335096225d2c0fa026014eb1b6b02aa01e598: Status 404 returned error can't find the container with id 079a99fde76d32d63207edf291e335096225d2c0fa026014eb1b6b02aa01e598 Nov 03 22:43:03.728044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728019 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728058 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728079 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728240 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728279 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728367 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728399 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728507 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333beb40-80d8-41d6-9d49-1308c7a6df9e-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.728702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.728678 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsfw\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-kube-api-access-sbsfw\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.829837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.829788 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.829837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.829837 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.829870 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.829907 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.829960 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333beb40-80d8-41d6-9d49-1308c7a6df9e-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830107 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsfw\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-kube-api-access-sbsfw\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830169 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830208 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830233 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830241 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-certs\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830302 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-credential-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830582 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830556 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-socket\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830564 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-data\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.830781 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.830757 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333beb40-80d8-41d6-9d49-1308c7a6df9e-istiod-ca-cert\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.832120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.832100 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-envoy\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.832588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.832571 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-podinfo\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.916744 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.916706 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-token\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:03.926840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:03.926777 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsfw\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-kube-api-access-sbsfw\") pod \"e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:04.115845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.095741 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:43:04.141213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.141131 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" event={"ID":"8f99857d-6f80-421c-b70d-5efb80a73712","Type":"ContainerStarted","Data":"079a99fde76d32d63207edf291e335096225d2c0fa026014eb1b6b02aa01e598"} Nov 03 22:43:04.143458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.143427 5272 generic.go:356] "Generic (PLEG): container finished" podID="ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" containerID="fb3b960c60460c3c5b5eea654f3a51dbd3e2f03c9a1b5a3a0e4dec948a6927d3" exitCode=0 Nov 03 22:43:04.143677 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.143508 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" event={"ID":"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc","Type":"ContainerDied","Data":"fb3b960c60460c3c5b5eea654f3a51dbd3e2f03c9a1b5a3a0e4dec948a6927d3"} Nov 03 22:43:04.204307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.204261 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:04.235256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.235225 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p"] Nov 03 22:43:04.241036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.241014 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.300490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.300378 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p"] Nov 03 22:43:04.338771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.338671 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.338771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.338734 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.338980 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.338891 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.339022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.338994 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4574947-0021-4d1e-9c38-b781ee531c1b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.339187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.339168 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.339226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.339215 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.339284 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.339268 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bk6l\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-kube-api-access-6bk6l\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.339338 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.339320 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.339371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.339352 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.463723 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.463762 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.463843 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bk6l\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-kube-api-access-6bk6l\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.463883 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.463899 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.463944 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.463965 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.464011 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.464076 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4574947-0021-4d1e-9c38-b781ee531c1b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.464734 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4574947-0021-4d1e-9c38-b781ee531c1b-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.468351 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.468926 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.474828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.469167 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.479045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.477296 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.479045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.477532 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.482047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.482024 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.559864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.559692 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.575859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.560418 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bk6l\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-kube-api-access-6bk6l\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:04.683774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.683743 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft"] Nov 03 22:43:04.686660 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:04.686623 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333beb40_80d8_41d6_9d49_1308c7a6df9e.slice/crio-7ec390069b6fb47943c8a32b9935de6b6a1a127096f9b0ce897e38723b5e40af WatchSource:0}: Error finding container 7ec390069b6fb47943c8a32b9935de6b6a1a127096f9b0ce897e38723b5e40af: Status 404 returned error can't find the container with id 7ec390069b6fb47943c8a32b9935de6b6a1a127096f9b0ce897e38723b5e40af Nov 03 22:43:04.689857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.689791 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:04.689945 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.689909 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:04.689986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.689963 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:04.871526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:04.871033 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:05.024942 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.002749 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6"] Nov 03 22:43:05.177831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.126393 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6"] Nov 03 22:43:05.177831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.126540 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251247 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251312 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251428 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251557 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rh8\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-kube-api-access-f6rh8\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251596 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251676 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251720 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251748 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.253831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.251777 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.377710 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.377772 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.377954 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.378106 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rh8\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-kube-api-access-f6rh8\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.378144 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.378227 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.378268 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.378298 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.378320 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.379466 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.397910 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.418312 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.432251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.430990 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.621588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.621501 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.782609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.666356 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.800827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.797407 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:05.950624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.850775 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rh8\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-kube-api-access-f6rh8\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:06.049868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.879665 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:06.050093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.952641 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:06.050566 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:05.991077 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:43:06.066401 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.006903 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:06.161932 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.149214 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:06.355468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.355423 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" event={"ID":"333beb40-80d8-41d6-9d49-1308c7a6df9e","Type":"ContainerStarted","Data":"425589bbab578f19f09ca9af43ff6a4878296347833a0ac2d8db14e570f5575e"} Nov 03 22:43:06.818331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.818285 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" event={"ID":"333beb40-80d8-41d6-9d49-1308c7a6df9e","Type":"ContainerStarted","Data":"7ec390069b6fb47943c8a32b9935de6b6a1a127096f9b0ce897e38723b5e40af"} Nov 03 22:43:06.821893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.391307 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.433169 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.484408 5272 generic.go:356] "Generic (PLEG): container finished" podID="8f99857d-6f80-421c-b70d-5efb80a73712" containerID="440bacf7145a44c2a66c7debf193fb61186ba4622d930151bf1d48d18c5f6f53" exitCode=0 Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.789899 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.791660 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.822970 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" event={"ID":"8f99857d-6f80-421c-b70d-5efb80a73712","Type":"ContainerDied","Data":"440bacf7145a44c2a66c7debf193fb61186ba4622d930151bf1d48d18c5f6f53"} Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.959412 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n"] Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.959460 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl"] Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.959660 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" containerID="cri-o://dcfb529d3145116cf3cd5f620d55083430c51f0c784a09767ff149f5daae5cd7" gracePeriod=30 Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.433123 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.825880 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podStartSLOduration=3.825861585 podStartE2EDuration="3.825861585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:06.771751581 +0000 UTC m=+1167.394059903" watchObservedRunningTime="2026-11-03 22:43:06.825861585 +0000 UTC m=+1167.448169895" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:06.963575 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" containerID="cri-o://6b3bcb90747eef4c07f9698286148121159fe0d0b9272372e22fa403bd1de2ac" gracePeriod=30 Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.069897 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.071793 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.123946 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:43:07.154606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.154201 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.181219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.181171 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.207094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.201124 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:07.207094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.203354 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:43:07.207094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.203415 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:43:07.236866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.213747 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:07.263902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.263753 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.263902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.263830 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:07.603359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.425087 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:07.603359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.425185 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:07.603359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.425233 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:07.734965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.670685 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:07.734965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.670758 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.865422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.827894 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.990926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.984936 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.990926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.985428 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:07.990926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:07.985747 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 03 22:43:08.026148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.025959 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.034153 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.033964 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.071254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.070878 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.295609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.093233 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.295609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.197425 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:43:08.295609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.197214 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:43:08.370062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.344549 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:08.370062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.344923 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 22:43:08.370062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.346533 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.370062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.346575 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:08.567344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.540677 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:43:08.567344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.540738 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:43:08.567344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.540778 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:43:08.600105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.269311 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.681833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.680841 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:08.681833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.680964 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:08.681833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.681016 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:08.839665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.839451 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:08.911618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.911040 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:43:08.911618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.911170 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:43:08.911618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.911229 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:43:08.911618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.911267 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:43:08.911618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.911320 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:43:08.930063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.913689 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" event={"ID":"b4574947-0021-4d1e-9c38-b781ee531c1b","Type":"ContainerStarted","Data":"a71dfcdae93fc2f48039a1fd777776e597ee8cc4d64dbfe71f5bd230d8f3f64e"} Nov 03 22:43:09.019551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.993037 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.019551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.993439 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 03 22:43:09.019551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:08.993698 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:43:09.019551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.007880 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p"] Nov 03 22:43:09.067694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.065903 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.067694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.066345 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:43:09.101889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.089183 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 22:43:09.101889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.093944 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.213931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.143892 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk"] Nov 03 22:43:09.213931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.176979 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef9fac-lsdmk"] Nov 03 22:43:09.255899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.239933 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:43:09.255899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.240528 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.321867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.277091 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:09.321867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.277153 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:09.414104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.413907 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:09.422183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.422076 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 22:43:09.564416 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.477206 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:09.951840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:09.948038 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:43:10.114834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.105061 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.301847 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.282218 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.305959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.305915 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:43:10.308938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.308903 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 22:43:10.309425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.306394 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:43:10.313941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.306423 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:43:10.451483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.451426 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 03 22:43:10.452095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.452051 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.496076 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.496021 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.578503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.578445 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:10.578788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.578757 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:10.636933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.636837 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.767746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.766953 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f6af3c-e6b8-412e-afad-296da72a59c7" path="/var/lib/kubelet/pods/b8f6af3c-e6b8-412e-afad-296da72a59c7/volumes" Nov 03 22:43:10.767746 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:43:10.767578 5272 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.28s" Nov 03 22:43:10.767746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.767663 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:43:10.767746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.767711 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:43:10.767746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.767743 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:43:10.768235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.767761 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv"] Nov 03 22:43:10.811460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.811402 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:10.813326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813058 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv"] Nov 03 22:43:10.813326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813118 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6"] Nov 03 22:43:10.813326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813182 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:43:10.813326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813221 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:43:10.813326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813254 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:43:10.813640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813352 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:43:10.813640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813399 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:43:10.813640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813434 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:43:10.813640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813464 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:43:10.813640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813496 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:43:10.813640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.813511 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" event={"ID":"b4574947-0021-4d1e-9c38-b781ee531c1b","Type":"ContainerStarted","Data":"b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9"} Nov 03 22:43:10.814134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.814116 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:10.817758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.814905 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:10.822819 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.822777 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" event={"ID":"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc","Type":"ContainerDied","Data":"9688b1d602fd80fa020a503b402e20bfb130cea9d53a7c83f6a96ed5ebced458"} Nov 03 22:43:10.822991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.822975 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9688b1d602fd80fa020a503b402e20bfb130cea9d53a7c83f6a96ed5ebced458" Nov 03 22:43:10.823303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.823283 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n" Nov 03 22:43:10.830541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.830509 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" event={"ID":"fb74ea1e-7dec-463e-9076-6fbedfb296ba","Type":"ContainerStarted","Data":"bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3"} Nov 03 22:43:10.830791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.830770 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" event={"ID":"fb74ea1e-7dec-463e-9076-6fbedfb296ba","Type":"ContainerStarted","Data":"dde924c957ffc1feddf2b52caf7e3c643ddd62c535c94e04c21f63080dbf5706"} Nov 03 22:43:10.895331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.895223 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd"] Nov 03 22:43:10.896054 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.896022 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:10.896191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.896099 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc8efed-t2dsd"] Nov 03 22:43:10.913903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.913865 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-xks-ca\") pod \"8f99857d-6f80-421c-b70d-5efb80a73712\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " Nov 03 22:43:10.914104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.913970 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-openshift-ca\") pod \"8f99857d-6f80-421c-b70d-5efb80a73712\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " Nov 03 22:43:10.914332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.914301 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-xks-ca\") pod \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " Nov 03 22:43:10.914504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.914479 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f99857d-6f80-421c-b70d-5efb80a73712-kube-api-access-ww92t\" (UniqueName: \"kubernetes.io/projected/8f99857d-6f80-421c-b70d-5efb80a73712-kube-api-access-ww92t\") pod \"8f99857d-6f80-421c-b70d-5efb80a73712\" (UID: \"8f99857d-6f80-421c-b70d-5efb80a73712\") " Nov 03 22:43:10.914660 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.914642 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-openshift-ca\") pod \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " Nov 03 22:43:10.914752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.914683 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-kube-api-access-ghk2r\" (UniqueName: \"kubernetes.io/projected/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-kube-api-access-ghk2r\") pod \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\" (UID: \"ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc\") " Nov 03 22:43:10.914854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.914825 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-openshift-ca" pod "8f99857d-6f80-421c-b70d-5efb80a73712" (UID: "8f99857d-6f80-421c-b70d-5efb80a73712"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:10.915334 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.915297 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:10.915615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.915591 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-openshift-ca" pod "ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" (UID: "ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:10.916065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.915764 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzfv\" (UniqueName: \"kubernetes.io/projected/80f1f1e7-665a-4a74-9a01-929c65938114-kube-api-access-qfzfv\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:10.916581 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.916549 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:10.917078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.917058 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:10.917223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.917199 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:10.928010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.927979 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-xks-ca" pod "8f99857d-6f80-421c-b70d-5efb80a73712" (UID: "8f99857d-6f80-421c-b70d-5efb80a73712"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:10.928258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.928198 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f99857d-6f80-421c-b70d-5efb80a73712-kube-api-access-ww92t" pod "8f99857d-6f80-421c-b70d-5efb80a73712" (UID: "8f99857d-6f80-421c-b70d-5efb80a73712"). InnerVolumeSpecName "kube-api-access-ww92t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:10.928363 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.928314 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-xks-ca" pod "ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" (UID: "ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:10.929469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:10.929410 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-kube-api-access-ghk2r" pod "ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" (UID: "ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc"). InnerVolumeSpecName "kube-api-access-ghk2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:11.018298 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.018266 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzfv\" (UniqueName: \"kubernetes.io/projected/80f1f1e7-665a-4a74-9a01-929c65938114-kube-api-access-qfzfv\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:11.019160 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.019129 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:11.019493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.019459 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:11.020033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.020013 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghk2r\" (UniqueName: \"kubernetes.io/projected/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-kube-api-access-ghk2r\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.020202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.020186 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f99857d-6f80-421c-b70d-5efb80a73712-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.020351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.020336 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.020505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.020487 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww92t\" (UniqueName: \"kubernetes.io/projected/8f99857d-6f80-421c-b70d-5efb80a73712-kube-api-access-ww92t\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.020613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.020530 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:11.028176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.028138 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:11.081660 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.081623 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzfv\" (UniqueName: \"kubernetes.io/projected/80f1f1e7-665a-4a74-9a01-929c65938114-kube-api-access-qfzfv\") pod \"maas-api-revoke-sub-5f08497c-dntnv\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:11.161244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.161149 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:11.161780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.161695 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:11.164591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.164548 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:11.164850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.164821 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:11.221201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.221158 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf"] Nov 03 22:43:11.222403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.222370 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" containerName="revoke-keys" Nov 03 22:43:11.224778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.222539 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" containerName="revoke-keys" Nov 03 22:43:11.224778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.222598 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8f99857d-6f80-421c-b70d-5efb80a73712" containerName="revoke-keys" Nov 03 22:43:11.224778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.222606 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f99857d-6f80-421c-b70d-5efb80a73712" containerName="revoke-keys" Nov 03 22:43:11.224778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.222785 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f99857d-6f80-421c-b70d-5efb80a73712" containerName="revoke-keys" Nov 03 22:43:11.224778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.222816 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" containerName="revoke-keys" Nov 03 22:43:11.279277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.244677 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:11.279277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.244742 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:11.382065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.382028 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf"] Nov 03 22:43:11.382392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.382326 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.459264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.459212 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4p76\" (UniqueName: \"kubernetes.io/projected/410d2ee2-649b-42c4-805e-e6e50d1a7968-kube-api-access-h4p76\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.459493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.459466 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.459564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.459509 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.501847 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.500223 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d85d05-9c2c-4c59-af24-b3e1cc67ec53" path="/var/lib/kubelet/pods/a4d85d05-9c2c-4c59-af24-b3e1cc67ec53/volumes" Nov 03 22:43:11.572423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.562761 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.572423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.562827 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.572423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.563020 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4p76\" (UniqueName: \"kubernetes.io/projected/410d2ee2-649b-42c4-805e-e6e50d1a7968-kube-api-access-h4p76\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.572423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.571740 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.592830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.582360 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.659181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.658945 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4p76\" (UniqueName: \"kubernetes.io/projected/410d2ee2-649b-42c4-805e-e6e50d1a7968-kube-api-access-h4p76\") pod \"maas-api-revoke-sub-a1ccd515-jpdwf\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.739888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.713790 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:11.923143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.890666 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:43:11.923143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:11.890732 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 22:43:12.172467 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:12.172430 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.276646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:12.276582 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:12.276939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:12.241475 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.277103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:12.277076 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:12.685740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:12.600615 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv"] Nov 03 22:43:12.999647 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:12.999550 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.282039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.274946 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 22:43:13.282039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.140005 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:43:13.282039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.137456 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.282039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.140047 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.282039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.246914 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.282039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.247220 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:43:13.371078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.357968 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.371078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.358046 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:13.427210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.427155 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.532952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.496814 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.532952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.498681 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.532952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.499049 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.532952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.499106 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:13.639882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.611995 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.667982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.667940 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.725829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.680794 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.856529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.845273 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mm95n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.856529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.845338 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:43:13.919148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.919112 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dvtvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.981833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:13.971375 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.053951 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m59rp" podUID="ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": context deadline exceeded" Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.187901 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.236042 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.235753 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" event={"ID":"7064709c-6647-4160-98ab-3801cdd1a76e","Type":"ContainerDied","Data":"6b3bcb90747eef4c07f9698286148121159fe0d0b9272372e22fa403bd1de2ac"} Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.236137 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf"] Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.205754 5272 generic.go:356] "Generic (PLEG): container finished" podID="7064709c-6647-4160-98ab-3801cdd1a76e" containerID="6b3bcb90747eef4c07f9698286148121159fe0d0b9272372e22fa403bd1de2ac" exitCode=0 Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.187950 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.236474 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.235414 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:14.236565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.236510 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:14.277929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.277884 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:14.278469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.278446 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:43:14.292447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.292407 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:14.293345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.293322 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:43:14.307963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.307938 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:43:14.321607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.321563 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:14.322277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.322252 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:43:14.340941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.340225 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:14.395374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.393775 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podStartSLOduration=10.393752635 podStartE2EDuration="10.393752635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:14.33947928 +0000 UTC m=+1174.961787583" watchObservedRunningTime="2026-11-03 22:43:14.393752635 +0000 UTC m=+1175.016060946" Nov 03 22:43:14.499660 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.499626 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:14.500392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.500371 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:43:14.514593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.513249 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" event={"ID":"8f99857d-6f80-421c-b70d-5efb80a73712","Type":"ContainerDied","Data":"079a99fde76d32d63207edf291e335096225d2c0fa026014eb1b6b02aa01e598"} Nov 03 22:43:14.514593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.513308 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="079a99fde76d32d63207edf291e335096225d2c0fa026014eb1b6b02aa01e598" Nov 03 22:43:14.514593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.513405 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm" Nov 03 22:43:14.524862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.518131 5272 generic.go:356] "Generic (PLEG): container finished" podID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerID="dcfb529d3145116cf3cd5f620d55083430c51f0c784a09767ff149f5daae5cd7" exitCode=0 Nov 03 22:43:14.525950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.525666 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" event={"ID":"e95a92be-6793-4ea9-be82-fdbeb4f54ca5","Type":"ContainerDied","Data":"dcfb529d3145116cf3cd5f620d55083430c51f0c784a09767ff149f5daae5cd7"} Nov 03 22:43:14.557109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.557047 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podStartSLOduration=10.557028194 podStartE2EDuration="10.557028194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:14.501676351 +0000 UTC m=+1175.123984661" watchObservedRunningTime="2026-11-03 22:43:14.557028194 +0000 UTC m=+1175.179336505" Nov 03 22:43:14.600117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.600027 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:43:14.637231 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.614920 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 22:43:14.637231 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.615351 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:43:14.637231 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.632718 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:43:14.668501 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.667954 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.727745 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-data\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.727854 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-kube-api-access-9ncxp\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-kube-api-access-9ncxp\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.727914 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-socket\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.727981 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-envoy\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.728011 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-podinfo\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.728139 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-credential-socket\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.728307 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-token\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.728366 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istiod-ca-cert\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.729327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.728415 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-certs\") pod \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\" (UID: \"e95a92be-6793-4ea9-be82-fdbeb4f54ca5\") " Nov 03 22:43:14.730837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.730422 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-data" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:14.736396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.732915 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:14.736774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.736754 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:43:14.738531 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.738348 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-token" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:14.738835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.738666 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-credential-socket" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:14.739978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.739721 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istiod-ca-cert" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:14.758077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.757965 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-certs" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:14.782381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.774726 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-socket" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:14.794015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.793971 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-envoy" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:14.822553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.822517 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:43:14.827662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.827634 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:43:14.831638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831593 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.831638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831627 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.831638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831642 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.831997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831656 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.831997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831668 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.831997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831679 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.831997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831693 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.831997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.831887 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-kube-api-access-9ncxp" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "kube-api-access-9ncxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:14.849179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.848727 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-podinfo" pod "e95a92be-6793-4ea9-be82-fdbeb4f54ca5" (UID: "e95a92be-6793-4ea9-be82-fdbeb4f54ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:14.875783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.874065 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:43:14.891530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.888728 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:43:14.891530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.891408 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:43:14.891530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.891456 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 22:43:14.935484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.934694 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ncxp\" (UniqueName: \"kubernetes.io/projected/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-kube-api-access-9ncxp\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:14.935484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:14.934733 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95a92be-6793-4ea9-be82-fdbeb4f54ca5-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.037007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.036972 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-envoy\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.037233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.037218 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-certs\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.037368 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.037354 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-data\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.037473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.037460 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-credential-socket\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.037560 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.037547 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-istio-token\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-istio-token\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.038034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.038012 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7064709c-6647-4160-98ab-3801cdd1a76e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064709c-6647-4160-98ab-3801cdd1a76e-istio-podinfo\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.038183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.038166 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7064709c-6647-4160-98ab-3801cdd1a76e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064709c-6647-4160-98ab-3801cdd1a76e-istiod-ca-cert\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.038864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.038312 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-certs" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:15.038946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.038354 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-credential-socket" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:15.038946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.038560 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-data" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:15.038946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.038840 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-kube-api-access-dtzkp\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-kube-api-access-dtzkp\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.039097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.039056 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-socket\") pod \"7064709c-6647-4160-98ab-3801cdd1a76e\" (UID: \"7064709c-6647-4160-98ab-3801cdd1a76e\") " Nov 03 22:43:15.039702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.039677 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.039797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.039707 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.039797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.039722 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.039993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.039936 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-socket" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:15.040232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.040208 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7064709c-6647-4160-98ab-3801cdd1a76e-istiod-ca-cert" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:15.041304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.041275 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7064709c-6647-4160-98ab-3801cdd1a76e-istio-podinfo" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:15.052424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.052209 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-envoy" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:15.052424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.052336 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-kube-api-access-dtzkp" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "kube-api-access-dtzkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:15.052424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.052394 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-istio-token" pod "7064709c-6647-4160-98ab-3801cdd1a76e" (UID: "7064709c-6647-4160-98ab-3801cdd1a76e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:15.141170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.141130 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.141170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.141168 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064709c-6647-4160-98ab-3801cdd1a76e-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.141377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.141183 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.141377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.141196 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064709c-6647-4160-98ab-3801cdd1a76e-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.141377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.141206 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064709c-6647-4160-98ab-3801cdd1a76e-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.141377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.141214 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzkp\" (UniqueName: \"kubernetes.io/projected/7064709c-6647-4160-98ab-3801cdd1a76e-kube-api-access-dtzkp\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:15.158456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.158326 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:15.158456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.158390 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:15.215034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.211910 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:15.215034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.211964 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 03 22:43:15.628429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.628389 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" event={"ID":"e95a92be-6793-4ea9-be82-fdbeb4f54ca5","Type":"ContainerDied","Data":"c015d3f35fcac43bfe13e5f3aa9572d317d8dfc97a5be5d199ee13f51d36ba73"} Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.690953 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" event={"ID":"7064709c-6647-4160-98ab-3801cdd1a76e","Type":"ContainerDied","Data":"eff1aa8261682231d0142e638e7298348bdf82732901a68036ebd43e15be0415"} Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.691035 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" event={"ID":"410d2ee2-649b-42c4-805e-e6e50d1a7968","Type":"ContainerStarted","Data":"c6fc10f0875cd9125843e54a2b0c77d899b63a61def4238a59a16dfa15ebee36"} Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.691080 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" event={"ID":"410d2ee2-649b-42c4-805e-e6e50d1a7968","Type":"ContainerStarted","Data":"422da3406a8dd3b97a792d37f02e75684bc92574401c95a35670b1c1e46c7644"} Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.634171 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl" Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.721861 5272 scope.go:122] "RemoveContainer" containerID="dcfb529d3145116cf3cd5f620d55083430c51f0c784a09767ff149f5daae5cd7" Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.640786 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n" Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.733095 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" event={"ID":"80f1f1e7-665a-4a74-9a01-929c65938114","Type":"ContainerStarted","Data":"09e36753d2a7a9d2329f4e95ece33e7009b4747fb42ae5c27fde56e38ca291d1"} Nov 03 22:43:15.734827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.733168 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" event={"ID":"80f1f1e7-665a-4a74-9a01-929c65938114","Type":"ContainerStarted","Data":"12560834488b6f1bf56cea6ac4cabc2461436b2cb45d84ab346a3f2c87a38414"} Nov 03 22:43:15.935079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.934976 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:43:15.935327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:15.935292 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 22:43:16.085829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.080095 5272 scope.go:122] "RemoveContainer" containerID="6b3bcb90747eef4c07f9698286148121159fe0d0b9272372e22fa403bd1de2ac" Nov 03 22:43:16.220648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.220466 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:16.220648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.220527 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:16.378645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.378580 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w"] Nov 03 22:43:16.380006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.379978 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" Nov 03 22:43:16.380156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.380142 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" Nov 03 22:43:16.380245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.380233 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" Nov 03 22:43:16.380354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.380339 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" Nov 03 22:43:16.380596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.380579 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" containerName="istio-proxy" Nov 03 22:43:16.380696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.380685 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" containerName="istio-proxy" Nov 03 22:43:16.391486 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.391459 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w"] Nov 03 22:43:16.391756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.391741 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.541311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.534572 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.541311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.534618 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfmz\" (UniqueName: \"kubernetes.io/projected/eb58de0e-903a-4dc0-b11b-c39bbafff353-kube-api-access-clfmz\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.541311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.534835 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.635704 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.635658 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.636179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.635950 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.636179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.635992 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfmz\" (UniqueName: \"kubernetes.io/projected/eb58de0e-903a-4dc0-b11b-c39bbafff353-kube-api-access-clfmz\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.637365 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.637313 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.692853 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.683855 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.739195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.735900 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfmz\" (UniqueName: \"kubernetes.io/projected/eb58de0e-903a-4dc0-b11b-c39bbafff353-kube-api-access-clfmz\") pod \"maas-api-revoke-sub-52dec9a1-vm27w\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.827994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.824433 5272 generic.go:356] "Generic (PLEG): container finished" podID="410d2ee2-649b-42c4-805e-e6e50d1a7968" containerID="c6fc10f0875cd9125843e54a2b0c77d899b63a61def4238a59a16dfa15ebee36" exitCode=0 Nov 03 22:43:16.827994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.824755 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" event={"ID":"410d2ee2-649b-42c4-805e-e6e50d1a7968","Type":"ContainerDied","Data":"c6fc10f0875cd9125843e54a2b0c77d899b63a61def4238a59a16dfa15ebee36"} Nov 03 22:43:16.827994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.827189 5272 generic.go:356] "Generic (PLEG): container finished" podID="80f1f1e7-665a-4a74-9a01-929c65938114" containerID="09e36753d2a7a9d2329f4e95ece33e7009b4747fb42ae5c27fde56e38ca291d1" exitCode=0 Nov 03 22:43:16.827994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.827278 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" event={"ID":"80f1f1e7-665a-4a74-9a01-929c65938114","Type":"ContainerDied","Data":"09e36753d2a7a9d2329f4e95ece33e7009b4747fb42ae5c27fde56e38ca291d1"} Nov 03 22:43:16.904121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.904064 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:43:16.904429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.904378 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 22:43:16.911087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.911049 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n"] Nov 03 22:43:16.917930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.917895 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mm95n"] Nov 03 22:43:16.932607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.932570 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:16.935443 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.935009 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:16.936326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.936302 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:16.941052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:16.940445 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:43:17.022972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.022903 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl"] Nov 03 22:43:17.037336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.037305 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dvtvl"] Nov 03 22:43:17.133638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.133576 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" podStartSLOduration=6.133557833 podStartE2EDuration="6.133557833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:17.128966957 +0000 UTC m=+1177.751275267" watchObservedRunningTime="2026-11-03 22:43:17.133557833 +0000 UTC m=+1177.755866143" Nov 03 22:43:17.196776 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.196728 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:17.200309 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.200259 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:17.300695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.300626 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" podStartSLOduration=10.300606983 podStartE2EDuration="10.300606983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:17.289610584 +0000 UTC m=+1177.911918893" watchObservedRunningTime="2026-11-03 22:43:17.300606983 +0000 UTC m=+1177.922915294" Nov 03 22:43:17.489286 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:17.482813 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb58de0e_903a_4dc0_b11b_c39bbafff353.slice/crio-51988325d68ea4500522fffcbf1d0fb9d3f019a46ddbf6d28996a4ca5cb3c4fe WatchSource:0}: Error finding container 51988325d68ea4500522fffcbf1d0fb9d3f019a46ddbf6d28996a4ca5cb3c4fe: Status 404 returned error can't find the container with id 51988325d68ea4500522fffcbf1d0fb9d3f019a46ddbf6d28996a4ca5cb3c4fe Nov 03 22:43:17.489286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.483967 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w"] Nov 03 22:43:17.504021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.503838 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7064709c-6647-4160-98ab-3801cdd1a76e" path="/var/lib/kubelet/pods/7064709c-6647-4160-98ab-3801cdd1a76e/volumes" Nov 03 22:43:17.505360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.504531 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95a92be-6793-4ea9-be82-fdbeb4f54ca5" path="/var/lib/kubelet/pods/e95a92be-6793-4ea9-be82-fdbeb4f54ca5/volumes" Nov 03 22:43:17.840553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.840507 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" event={"ID":"eb58de0e-903a-4dc0-b11b-c39bbafff353","Type":"ContainerStarted","Data":"8999cffc30d2ed35822065bca1ab7a7a9faed9bb933f3037d2bb59f050a86066"} Nov 03 22:43:17.841007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.840607 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" event={"ID":"eb58de0e-903a-4dc0-b11b-c39bbafff353","Type":"ContainerStarted","Data":"51988325d68ea4500522fffcbf1d0fb9d3f019a46ddbf6d28996a4ca5cb3c4fe"} Nov 03 22:43:17.871979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.871939 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 22:43:17.872155 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.871997 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 22:43:17.897847 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.897729 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45"] Nov 03 22:43:17.902780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.902752 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:17.911687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.910361 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:17.911687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.910675 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tcl\" (UniqueName: \"kubernetes.io/projected/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-kube-api-access-f9tcl\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:17.911687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.910710 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:17.972658 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.972561 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45"] Nov 03 22:43:17.975461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:17.975394 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" podStartSLOduration=1.975374096 podStartE2EDuration="1.975374096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:17.958284206 +0000 UTC m=+1178.580592528" watchObservedRunningTime="2026-11-03 22:43:17.975374096 +0000 UTC m=+1178.597682405" Nov 03 22:43:18.017172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.015488 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:18.017172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.015658 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tcl\" (UniqueName: \"kubernetes.io/projected/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-kube-api-access-f9tcl\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:18.017172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.015680 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:18.017172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.016441 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:18.021448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.021390 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:18.074568 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.074540 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tcl\" (UniqueName: \"kubernetes.io/projected/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-kube-api-access-f9tcl\") pod \"maas-api-revoke-sub-c8e0f210-whm45\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:18.091996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.091888 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:18.096227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.096202 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:18.158712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.158679 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:43:18.158876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.158737 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:43:18.218862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.218823 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-openshift-ca\") pod \"80f1f1e7-665a-4a74-9a01-929c65938114\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " Nov 03 22:43:18.219043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.218879 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-xks-ca\") pod \"410d2ee2-649b-42c4-805e-e6e50d1a7968\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " Nov 03 22:43:18.219043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.218945 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/410d2ee2-649b-42c4-805e-e6e50d1a7968-kube-api-access-h4p76\" (UniqueName: \"kubernetes.io/projected/410d2ee2-649b-42c4-805e-e6e50d1a7968-kube-api-access-h4p76\") pod \"410d2ee2-649b-42c4-805e-e6e50d1a7968\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " Nov 03 22:43:18.219043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.219034 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-openshift-ca\") pod \"410d2ee2-649b-42c4-805e-e6e50d1a7968\" (UID: \"410d2ee2-649b-42c4-805e-e6e50d1a7968\") " Nov 03 22:43:18.219214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.219182 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-xks-ca\") pod \"80f1f1e7-665a-4a74-9a01-929c65938114\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " Nov 03 22:43:18.219276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.219233 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f1f1e7-665a-4a74-9a01-929c65938114-kube-api-access-qfzfv\" (UniqueName: \"kubernetes.io/projected/80f1f1e7-665a-4a74-9a01-929c65938114-kube-api-access-qfzfv\") pod \"80f1f1e7-665a-4a74-9a01-929c65938114\" (UID: \"80f1f1e7-665a-4a74-9a01-929c65938114\") " Nov 03 22:43:18.219276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.219244 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-openshift-ca" pod "80f1f1e7-665a-4a74-9a01-929c65938114" (UID: "80f1f1e7-665a-4a74-9a01-929c65938114"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:18.220006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.219726 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-openshift-ca" pod "410d2ee2-649b-42c4-805e-e6e50d1a7968" (UID: "410d2ee2-649b-42c4-805e-e6e50d1a7968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:18.220357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.220336 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:18.220491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.220475 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:18.222065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.222032 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-xks-ca" pod "410d2ee2-649b-42c4-805e-e6e50d1a7968" (UID: "410d2ee2-649b-42c4-805e-e6e50d1a7968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:18.222488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.222457 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410d2ee2-649b-42c4-805e-e6e50d1a7968-kube-api-access-h4p76" pod "410d2ee2-649b-42c4-805e-e6e50d1a7968" (UID: "410d2ee2-649b-42c4-805e-e6e50d1a7968"). InnerVolumeSpecName "kube-api-access-h4p76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:18.222651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.222625 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-xks-ca" pod "80f1f1e7-665a-4a74-9a01-929c65938114" (UID: "80f1f1e7-665a-4a74-9a01-929c65938114"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:18.222866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.222846 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f1f1e7-665a-4a74-9a01-929c65938114-kube-api-access-qfzfv" pod "80f1f1e7-665a-4a74-9a01-929c65938114" (UID: "80f1f1e7-665a-4a74-9a01-929c65938114"). InnerVolumeSpecName "kube-api-access-qfzfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:18.235142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.235110 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:18.330163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.330125 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80f1f1e7-665a-4a74-9a01-929c65938114-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:18.330304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.330167 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzfv\" (UniqueName: \"kubernetes.io/projected/80f1f1e7-665a-4a74-9a01-929c65938114-kube-api-access-qfzfv\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:18.330304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.330184 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410d2ee2-649b-42c4-805e-e6e50d1a7968-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:18.330304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.330197 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4p76\" (UniqueName: \"kubernetes.io/projected/410d2ee2-649b-42c4-805e-e6e50d1a7968-kube-api-access-h4p76\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:18.651900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.651766 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45"] Nov 03 22:43:18.665937 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:18.665899 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9ba3d8_f5b1_4253_88b5_5bd31da31100.slice/crio-5da8d060b474f5a20ae97ceb62a5462390f07f22082d579a33bb13374f183dec WatchSource:0}: Error finding container 5da8d060b474f5a20ae97ceb62a5462390f07f22082d579a33bb13374f183dec: Status 404 returned error can't find the container with id 5da8d060b474f5a20ae97ceb62a5462390f07f22082d579a33bb13374f183dec Nov 03 22:43:18.858828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.853483 5272 generic.go:356] "Generic (PLEG): container finished" podID="eb58de0e-903a-4dc0-b11b-c39bbafff353" containerID="8999cffc30d2ed35822065bca1ab7a7a9faed9bb933f3037d2bb59f050a86066" exitCode=0 Nov 03 22:43:18.858828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.853602 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" event={"ID":"eb58de0e-903a-4dc0-b11b-c39bbafff353","Type":"ContainerDied","Data":"8999cffc30d2ed35822065bca1ab7a7a9faed9bb933f3037d2bb59f050a86066"} Nov 03 22:43:18.858828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.855584 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" event={"ID":"0c9ba3d8-f5b1-4253-88b5-5bd31da31100","Type":"ContainerStarted","Data":"5da8d060b474f5a20ae97ceb62a5462390f07f22082d579a33bb13374f183dec"} Nov 03 22:43:18.858828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.857254 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" event={"ID":"410d2ee2-649b-42c4-805e-e6e50d1a7968","Type":"ContainerDied","Data":"422da3406a8dd3b97a792d37f02e75684bc92574401c95a35670b1c1e46c7644"} Nov 03 22:43:18.858828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.857288 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422da3406a8dd3b97a792d37f02e75684bc92574401c95a35670b1c1e46c7644" Nov 03 22:43:18.858828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.857343 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf" Nov 03 22:43:18.863845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.860962 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" Nov 03 22:43:18.863845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.861022 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv" event={"ID":"80f1f1e7-665a-4a74-9a01-929c65938114","Type":"ContainerDied","Data":"12560834488b6f1bf56cea6ac4cabc2461436b2cb45d84ab346a3f2c87a38414"} Nov 03 22:43:18.863845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.861072 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12560834488b6f1bf56cea6ac4cabc2461436b2cb45d84ab346a3f2c87a38414" Nov 03 22:43:18.988896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.987428 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:18.988896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.988726 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:18.991283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:18.991229 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:43:19.162791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:19.162756 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:19.163103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:19.163086 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:19.163854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:19.163834 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:43:19.872875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:19.872830 5272 generic.go:356] "Generic (PLEG): container finished" podID="0c9ba3d8-f5b1-4253-88b5-5bd31da31100" containerID="4c99c0692b3ffe5753cfc8e51b4e399f6dcdad106d1d8ee80375acbec168b980" exitCode=0 Nov 03 22:43:19.873304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:19.872922 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" event={"ID":"0c9ba3d8-f5b1-4253-88b5-5bd31da31100","Type":"ContainerDied","Data":"4c99c0692b3ffe5753cfc8e51b4e399f6dcdad106d1d8ee80375acbec168b980"} Nov 03 22:43:20.037708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.037673 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:20.054896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.054859 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-openshift-ca\") pod \"eb58de0e-903a-4dc0-b11b-c39bbafff353\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " Nov 03 22:43:20.055065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.054937 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-xks-ca\") pod \"eb58de0e-903a-4dc0-b11b-c39bbafff353\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " Nov 03 22:43:20.055188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.055171 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb58de0e-903a-4dc0-b11b-c39bbafff353-kube-api-access-clfmz\" (UniqueName: \"kubernetes.io/projected/eb58de0e-903a-4dc0-b11b-c39bbafff353-kube-api-access-clfmz\") pod \"eb58de0e-903a-4dc0-b11b-c39bbafff353\" (UID: \"eb58de0e-903a-4dc0-b11b-c39bbafff353\") " Nov 03 22:43:20.055266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.055243 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-openshift-ca" pod "eb58de0e-903a-4dc0-b11b-c39bbafff353" (UID: "eb58de0e-903a-4dc0-b11b-c39bbafff353"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:20.055662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.055641 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:20.057090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.057063 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-xks-ca" pod "eb58de0e-903a-4dc0-b11b-c39bbafff353" (UID: "eb58de0e-903a-4dc0-b11b-c39bbafff353"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:20.057208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.057184 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb58de0e-903a-4dc0-b11b-c39bbafff353-kube-api-access-clfmz" pod "eb58de0e-903a-4dc0-b11b-c39bbafff353" (UID: "eb58de0e-903a-4dc0-b11b-c39bbafff353"). InnerVolumeSpecName "kube-api-access-clfmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:20.156882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.156848 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfmz\" (UniqueName: \"kubernetes.io/projected/eb58de0e-903a-4dc0-b11b-c39bbafff353-kube-api-access-clfmz\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:20.156882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.156882 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb58de0e-903a-4dc0-b11b-c39bbafff353-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:20.882794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.882762 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" Nov 03 22:43:20.883210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.882826 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w" event={"ID":"eb58de0e-903a-4dc0-b11b-c39bbafff353","Type":"ContainerDied","Data":"51988325d68ea4500522fffcbf1d0fb9d3f019a46ddbf6d28996a4ca5cb3c4fe"} Nov 03 22:43:20.883210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:20.882883 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51988325d68ea4500522fffcbf1d0fb9d3f019a46ddbf6d28996a4ca5cb3c4fe" Nov 03 22:43:21.100076 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.100041 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:21.180167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.180069 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-xks-ca\") pod \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " Nov 03 22:43:21.180167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.180115 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-kube-api-access-f9tcl\" (UniqueName: \"kubernetes.io/projected/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-kube-api-access-f9tcl\") pod \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " Nov 03 22:43:21.180423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.180169 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-openshift-ca\") pod \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\" (UID: \"0c9ba3d8-f5b1-4253-88b5-5bd31da31100\") " Nov 03 22:43:21.180570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.180533 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-openshift-ca" pod "0c9ba3d8-f5b1-4253-88b5-5bd31da31100" (UID: "0c9ba3d8-f5b1-4253-88b5-5bd31da31100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:21.180722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.180705 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:21.182304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.182280 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-xks-ca" pod "0c9ba3d8-f5b1-4253-88b5-5bd31da31100" (UID: "0c9ba3d8-f5b1-4253-88b5-5bd31da31100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:21.182385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.182350 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-kube-api-access-f9tcl" pod "0c9ba3d8-f5b1-4253-88b5-5bd31da31100" (UID: "0c9ba3d8-f5b1-4253-88b5-5bd31da31100"). InnerVolumeSpecName "kube-api-access-f9tcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:21.282717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.282676 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:21.282934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.282737 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9tcl\" (UniqueName: \"kubernetes.io/projected/0c9ba3d8-f5b1-4253-88b5-5bd31da31100-kube-api-access-f9tcl\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:21.892656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.892624 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" Nov 03 22:43:21.892656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.892647 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45" event={"ID":"0c9ba3d8-f5b1-4253-88b5-5bd31da31100","Type":"ContainerDied","Data":"5da8d060b474f5a20ae97ceb62a5462390f07f22082d579a33bb13374f183dec"} Nov 03 22:43:21.893192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.892701 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5da8d060b474f5a20ae97ceb62a5462390f07f22082d579a33bb13374f183dec" Nov 03 22:43:21.995556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.995520 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz"] Nov 03 22:43:21.996247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996227 5272 cpu_manager.go:414] "containerMap: removing container" podUID="eb58de0e-903a-4dc0-b11b-c39bbafff353" containerName="revoke-keys" Nov 03 22:43:21.996247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996247 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb58de0e-903a-4dc0-b11b-c39bbafff353" containerName="revoke-keys" Nov 03 22:43:21.996421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996271 5272 cpu_manager.go:414] "containerMap: removing container" podUID="80f1f1e7-665a-4a74-9a01-929c65938114" containerName="revoke-keys" Nov 03 22:43:21.996421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996279 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f1f1e7-665a-4a74-9a01-929c65938114" containerName="revoke-keys" Nov 03 22:43:21.996421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996295 5272 cpu_manager.go:414] "containerMap: removing container" podUID="410d2ee2-649b-42c4-805e-e6e50d1a7968" containerName="revoke-keys" Nov 03 22:43:21.996421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996302 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="410d2ee2-649b-42c4-805e-e6e50d1a7968" containerName="revoke-keys" Nov 03 22:43:21.996421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996331 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0c9ba3d8-f5b1-4253-88b5-5bd31da31100" containerName="revoke-keys" Nov 03 22:43:21.996421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996337 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9ba3d8-f5b1-4253-88b5-5bd31da31100" containerName="revoke-keys" Nov 03 22:43:21.996727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996490 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb58de0e-903a-4dc0-b11b-c39bbafff353" containerName="revoke-keys" Nov 03 22:43:21.996727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996503 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f1f1e7-665a-4a74-9a01-929c65938114" containerName="revoke-keys" Nov 03 22:43:21.996727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996516 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="410d2ee2-649b-42c4-805e-e6e50d1a7968" containerName="revoke-keys" Nov 03 22:43:21.996727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:21.996529 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9ba3d8-f5b1-4253-88b5-5bd31da31100" containerName="revoke-keys" Nov 03 22:43:22.000503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.000478 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.040642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.040612 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz"] Nov 03 22:43:22.053510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.053477 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz"] Nov 03 22:43:22.059905 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.059874 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5812b5d5-m5pzz"] Nov 03 22:43:22.081934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.081902 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:43:22.082112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.081971 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:43:22.082112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.081978 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:22.092207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.092174 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:43:22.092903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.092882 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.092996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.092950 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwrm\" (UniqueName: \"kubernetes.io/projected/d5509e52-244a-4f7c-a0c4-7313e566acee-kube-api-access-lvwrm\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.093119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.093104 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.194657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.194551 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.194842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.194749 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.194928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.194855 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwrm\" (UniqueName: \"kubernetes.io/projected/d5509e52-244a-4f7c-a0c4-7313e566acee-kube-api-access-lvwrm\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.195321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.195297 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.197124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.197100 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.252215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.252181 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwrm\" (UniqueName: \"kubernetes.io/projected/d5509e52-244a-4f7c-a0c4-7313e566acee-kube-api-access-lvwrm\") pod \"maas-api-revoke-sub-0dcee50e-xm9bz\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.316273 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.316240 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:22.714824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.714764 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz"] Nov 03 22:43:22.720212 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:22.720179 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5509e52_244a_4f7c_a0c4_7313e566acee.slice/crio-0f0effb205691801fd53429864d4d705db59d014f87dc1354edf77a017b34ca6 WatchSource:0}: Error finding container 0f0effb205691801fd53429864d4d705db59d014f87dc1354edf77a017b34ca6: Status 404 returned error can't find the container with id 0f0effb205691801fd53429864d4d705db59d014f87dc1354edf77a017b34ca6 Nov 03 22:43:22.910442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:22.910398 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" event={"ID":"d5509e52-244a-4f7c-a0c4-7313e566acee","Type":"ContainerStarted","Data":"0f0effb205691801fd53429864d4d705db59d014f87dc1354edf77a017b34ca6"} Nov 03 22:43:23.493955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:23.493918 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73c8102-dfd3-46cd-82cc-b7bb1b31e99e" path="/var/lib/kubelet/pods/d73c8102-dfd3-46cd-82cc-b7bb1b31e99e/volumes" Nov 03 22:43:23.920245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:23.920205 5272 generic.go:356] "Generic (PLEG): container finished" podID="d5509e52-244a-4f7c-a0c4-7313e566acee" containerID="3d1b8a4420d3d86bb7a1252f000a135a34f078c88b5250bdf1bbc7da8a32ee79" exitCode=0 Nov 03 22:43:23.920637 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:23.920284 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" event={"ID":"d5509e52-244a-4f7c-a0c4-7313e566acee","Type":"ContainerDied","Data":"3d1b8a4420d3d86bb7a1252f000a135a34f078c88b5250bdf1bbc7da8a32ee79"} Nov 03 22:43:24.078850 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:24.078813 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p"] Nov 03 22:43:24.082668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:24.082644 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c333ad2-h758p"] Nov 03 22:43:24.501972 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:43:24.501934 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:43:25.083246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.083221 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:25.135650 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.135613 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5509e52-244a-4f7c-a0c4-7313e566acee-kube-api-access-lvwrm\" (UniqueName: \"kubernetes.io/projected/d5509e52-244a-4f7c-a0c4-7313e566acee-kube-api-access-lvwrm\") pod \"d5509e52-244a-4f7c-a0c4-7313e566acee\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " Nov 03 22:43:25.135912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.135760 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-openshift-ca\") pod \"d5509e52-244a-4f7c-a0c4-7313e566acee\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " Nov 03 22:43:25.136060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.136032 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-xks-ca\") pod \"d5509e52-244a-4f7c-a0c4-7313e566acee\" (UID: \"d5509e52-244a-4f7c-a0c4-7313e566acee\") " Nov 03 22:43:25.136216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.136191 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-openshift-ca" pod "d5509e52-244a-4f7c-a0c4-7313e566acee" (UID: "d5509e52-244a-4f7c-a0c4-7313e566acee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:25.136685 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.136667 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:25.138102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.138073 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-xks-ca" pod "d5509e52-244a-4f7c-a0c4-7313e566acee" (UID: "d5509e52-244a-4f7c-a0c4-7313e566acee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:25.138172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.138078 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5509e52-244a-4f7c-a0c4-7313e566acee-kube-api-access-lvwrm" pod "d5509e52-244a-4f7c-a0c4-7313e566acee" (UID: "d5509e52-244a-4f7c-a0c4-7313e566acee"). InnerVolumeSpecName "kube-api-access-lvwrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:25.238593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.238496 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvwrm\" (UniqueName: \"kubernetes.io/projected/d5509e52-244a-4f7c-a0c4-7313e566acee-kube-api-access-lvwrm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:25.238593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.238530 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5509e52-244a-4f7c-a0c4-7313e566acee-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:25.494069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.493987 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423" path="/var/lib/kubelet/pods/1f7cdb9e-c2e0-4ae9-9b13-239dc8a86423/volumes" Nov 03 22:43:25.967827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.966079 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" event={"ID":"d5509e52-244a-4f7c-a0c4-7313e566acee","Type":"ContainerDied","Data":"0f0effb205691801fd53429864d4d705db59d014f87dc1354edf77a017b34ca6"} Nov 03 22:43:25.967827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.966192 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0effb205691801fd53429864d4d705db59d014f87dc1354edf77a017b34ca6" Nov 03 22:43:25.967827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:25.966276 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz" Nov 03 22:43:28.321002 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.320969 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2"] Nov 03 22:43:28.321783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.321757 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d5509e52-244a-4f7c-a0c4-7313e566acee" containerName="revoke-keys" Nov 03 22:43:28.321783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.321782 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5509e52-244a-4f7c-a0c4-7313e566acee" containerName="revoke-keys" Nov 03 22:43:28.321991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.321978 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5509e52-244a-4f7c-a0c4-7313e566acee" containerName="revoke-keys" Nov 03 22:43:28.325599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.325571 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.489758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.489717 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.489947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.489891 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.490024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.490001 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.490069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.490053 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.490204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.490183 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt99n\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-kube-api-access-zt99n\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.490264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.490242 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.490395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.490372 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.490521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.490488 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.490666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.490554 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.527653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.527615 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2"] Nov 03 22:43:28.592094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.591999 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt99n\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-kube-api-access-zt99n\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592048 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592099 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592152 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592185 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592226 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592281 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592358 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592417 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592608 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-credential-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592632 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-certs\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592844 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-data\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.592889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.592849 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-socket\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.593190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.593171 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istiod-ca-cert\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.594540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.594517 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-envoy\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.594844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.594825 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-podinfo\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.643106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.643075 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg"] Nov 03 22:43:28.649016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.648992 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.650553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.650526 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-token\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.650657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.650530 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt99n\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-kube-api-access-zt99n\") pod \"e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.698008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.697978 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:43:28.698266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.698214 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:43:28.698266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.698224 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:43:28.698452 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.698274 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:28.708715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.708677 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg"] Nov 03 22:43:28.794673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.794637 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfm7x\" (UniqueName: \"kubernetes.io/projected/c45a9146-b621-4b7a-9522-3f5b65513845-kube-api-access-zfm7x\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.794930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.794905 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.795069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.795046 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.896489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.896402 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.896643 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.896519 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.896643 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.896564 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfm7x\" (UniqueName: \"kubernetes.io/projected/c45a9146-b621-4b7a-9522-3f5b65513845-kube-api-access-zfm7x\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.897224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.897199 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.898872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.898851 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.940225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.940193 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:28.957960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.957575 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfm7x\" (UniqueName: \"kubernetes.io/projected/c45a9146-b621-4b7a-9522-3f5b65513845-kube-api-access-zfm7x\") pod \"maas-api-revoke-sub-8b2a8438-s85hg\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:28.977357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:28.977320 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:29.433705 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:29.433644 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg"] Nov 03 22:43:29.439033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:29.439003 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2"] Nov 03 22:43:29.443086 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:29.443062 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45a9146_b621_4b7a_9522_3f5b65513845.slice/crio-e8e231d0d84bf1e188c9e53abd67d37ca580164a008c3eea98f39cc3f3ff1d0e WatchSource:0}: Error finding container e8e231d0d84bf1e188c9e53abd67d37ca580164a008c3eea98f39cc3f3ff1d0e: Status 404 returned error can't find the container with id e8e231d0d84bf1e188c9e53abd67d37ca580164a008c3eea98f39cc3f3ff1d0e Nov 03 22:43:29.484161 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:29.484132 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa6e3bc_8f1e_4d02_923d_16e487dd860f.slice/crio-dd4c516450d0473d0752e0ac0ebb7529a15eb76ef955db35cafdef946df13670 WatchSource:0}: Error finding container dd4c516450d0473d0752e0ac0ebb7529a15eb76ef955db35cafdef946df13670: Status 404 returned error can't find the container with id dd4c516450d0473d0752e0ac0ebb7529a15eb76ef955db35cafdef946df13670 Nov 03 22:43:29.488793 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:29.488751 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.488906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:29.488889 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:29.488946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:29.488935 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:30.012011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.005844 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" event={"ID":"c45a9146-b621-4b7a-9522-3f5b65513845","Type":"ContainerStarted","Data":"23819f23ec4170e098e0f0394c5d672987ced3ccb2badf5fa17e8ff177113459"} Nov 03 22:43:30.012011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.005912 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" event={"ID":"c45a9146-b621-4b7a-9522-3f5b65513845","Type":"ContainerStarted","Data":"e8e231d0d84bf1e188c9e53abd67d37ca580164a008c3eea98f39cc3f3ff1d0e"} Nov 03 22:43:30.012011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.009346 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" event={"ID":"7fa6e3bc-8f1e-4d02-923d-16e487dd860f","Type":"ContainerStarted","Data":"e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51"} Nov 03 22:43:30.012011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.009400 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" event={"ID":"7fa6e3bc-8f1e-4d02-923d-16e487dd860f","Type":"ContainerStarted","Data":"dd4c516450d0473d0752e0ac0ebb7529a15eb76ef955db35cafdef946df13670"} Nov 03 22:43:30.120895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.120834 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" podStartSLOduration=2.120814493 podStartE2EDuration="2.120814493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:30.117167393 +0000 UTC m=+1190.739475702" watchObservedRunningTime="2026-11-03 22:43:30.120814493 +0000 UTC m=+1190.743122796" Nov 03 22:43:30.235905 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.235848 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podStartSLOduration=2.235832661 podStartE2EDuration="2.235832661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:30.232102204 +0000 UTC m=+1190.854410513" watchObservedRunningTime="2026-11-03 22:43:30.235832661 +0000 UTC m=+1190.858140959" Nov 03 22:43:30.940520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.940490 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:30.942324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.942297 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:43:30.942474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:30.942355 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 22:43:31.076839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:31.074743 5272 generic.go:356] "Generic (PLEG): container finished" podID="c45a9146-b621-4b7a-9522-3f5b65513845" containerID="23819f23ec4170e098e0f0394c5d672987ced3ccb2badf5fa17e8ff177113459" exitCode=0 Nov 03 22:43:31.076839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:31.076229 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" event={"ID":"c45a9146-b621-4b7a-9522-3f5b65513845","Type":"ContainerDied","Data":"23819f23ec4170e098e0f0394c5d672987ced3ccb2badf5fa17e8ff177113459"} Nov 03 22:43:31.941425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:31.941393 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:43:31.941831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:31.941446 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 22:43:32.290596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.290569 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:32.470243 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.470193 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-xks-ca\") pod \"c45a9146-b621-4b7a-9522-3f5b65513845\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " Nov 03 22:43:32.470243 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.470249 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-openshift-ca\") pod \"c45a9146-b621-4b7a-9522-3f5b65513845\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " Nov 03 22:43:32.470523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.470285 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45a9146-b621-4b7a-9522-3f5b65513845-kube-api-access-zfm7x\" (UniqueName: \"kubernetes.io/projected/c45a9146-b621-4b7a-9522-3f5b65513845-kube-api-access-zfm7x\") pod \"c45a9146-b621-4b7a-9522-3f5b65513845\" (UID: \"c45a9146-b621-4b7a-9522-3f5b65513845\") " Nov 03 22:43:32.470728 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.470700 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-openshift-ca" pod "c45a9146-b621-4b7a-9522-3f5b65513845" (UID: "c45a9146-b621-4b7a-9522-3f5b65513845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:32.471383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.471355 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:32.472891 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.472861 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45a9146-b621-4b7a-9522-3f5b65513845-kube-api-access-zfm7x" pod "c45a9146-b621-4b7a-9522-3f5b65513845" (UID: "c45a9146-b621-4b7a-9522-3f5b65513845"). InnerVolumeSpecName "kube-api-access-zfm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:32.473075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.473046 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-xks-ca" pod "c45a9146-b621-4b7a-9522-3f5b65513845" (UID: "c45a9146-b621-4b7a-9522-3f5b65513845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:32.573159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.573121 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfm7x\" (UniqueName: \"kubernetes.io/projected/c45a9146-b621-4b7a-9522-3f5b65513845-kube-api-access-zfm7x\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:32.573159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.573156 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45a9146-b621-4b7a-9522-3f5b65513845-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:32.941625 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.941537 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:43:32.941625 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:32.941595 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 22:43:33.093880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:33.093840 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" event={"ID":"c45a9146-b621-4b7a-9522-3f5b65513845","Type":"ContainerDied","Data":"e8e231d0d84bf1e188c9e53abd67d37ca580164a008c3eea98f39cc3f3ff1d0e"} Nov 03 22:43:33.094070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:33.093892 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e231d0d84bf1e188c9e53abd67d37ca580164a008c3eea98f39cc3f3ff1d0e" Nov 03 22:43:33.094070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:33.093892 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg" Nov 03 22:43:33.941301 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:33.941266 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:43:33.941488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:33.941329 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 22:43:34.810495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.810464 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh"] Nov 03 22:43:34.811108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.811091 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c45a9146-b621-4b7a-9522-3f5b65513845" containerName="revoke-keys" Nov 03 22:43:34.811108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.811108 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45a9146-b621-4b7a-9522-3f5b65513845" containerName="revoke-keys" Nov 03 22:43:34.811242 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.811232 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45a9146-b621-4b7a-9522-3f5b65513845" containerName="revoke-keys" Nov 03 22:43:34.814812 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.814785 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:34.917351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.917321 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh"] Nov 03 22:43:34.940745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.940708 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:43:34.940950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:34.940777 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 22:43:35.006813 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.006762 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.006970 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.006913 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr78t\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-kube-api-access-zr78t\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.007038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.006970 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.007084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.007058 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.007135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.007094 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c945d247-a300-4b2c-bf34-a46626323f6e-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.007135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.007110 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.007271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.007248 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c945d247-a300-4b2c-bf34-a46626323f6e-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.007310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.007303 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.007441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.007429 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112242 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112327 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112405 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr78t\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-kube-api-access-zr78t\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112476 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112519 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112562 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c945d247-a300-4b2c-bf34-a46626323f6e-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112590 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112682 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c945d247-a300-4b2c-bf34-a46626323f6e-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.112778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.112752 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.113411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.113384 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-certs\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.113483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.113432 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-data\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.113766 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.113727 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-credential-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.119490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.114046 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-socket\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.119490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.115269 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c945d247-a300-4b2c-bf34-a46626323f6e-istiod-ca-cert\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.119490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.117004 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-envoy\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.122854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.122797 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c945d247-a300-4b2c-bf34-a46626323f6e-istio-podinfo\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.172788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.172748 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr78t\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-kube-api-access-zr78t\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.173358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.173328 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-istio-token\") pod \"e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.430076 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.429985 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:35.880997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.880963 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh"] Nov 03 22:43:35.885243 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:35.885212 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc945d247_a300_4b2c_bf34_a46626323f6e.slice/crio-99ee8fcd3dc531f0e7a4f8d72d10769ffda20634b264be78fd477a162b63d1c9 WatchSource:0}: Error finding container 99ee8fcd3dc531f0e7a4f8d72d10769ffda20634b264be78fd477a162b63d1c9: Status 404 returned error can't find the container with id 99ee8fcd3dc531f0e7a4f8d72d10769ffda20634b264be78fd477a162b63d1c9 Nov 03 22:43:35.888037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.888003 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.888099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.888071 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.888142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.888101 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:35.941332 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.941290 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 22:43:35.941482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:35.941346 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 22:43:36.129074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.129034 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" event={"ID":"c945d247-a300-4b2c-bf34-a46626323f6e","Type":"ContainerStarted","Data":"f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24"} Nov 03 22:43:36.129327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.129309 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" event={"ID":"c945d247-a300-4b2c-bf34-a46626323f6e","Type":"ContainerStarted","Data":"99ee8fcd3dc531f0e7a4f8d72d10769ffda20634b264be78fd477a162b63d1c9"} Nov 03 22:43:36.246879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.246437 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podStartSLOduration=2.246419431 podStartE2EDuration="2.246419431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:36.242167377 +0000 UTC m=+1196.864475698" watchObservedRunningTime="2026-11-03 22:43:36.246419431 +0000 UTC m=+1196.868727740" Nov 03 22:43:36.295966 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.295930 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn"] Nov 03 22:43:36.300456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.300432 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.353833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.351091 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:43:36.362119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.360189 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:43:36.362119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.360986 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:36.362119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.361073 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:43:36.383017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.381077 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn"] Nov 03 22:43:36.463652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.462767 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.463652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.463015 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.463652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.463288 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46m8x\" (UniqueName: \"kubernetes.io/projected/75439837-68b1-4d0b-8fbc-df6bf3108412-kube-api-access-46m8x\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.469592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.467130 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:36.471624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.471596 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:36.471783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.471663 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:36.565367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.565327 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46m8x\" (UniqueName: \"kubernetes.io/projected/75439837-68b1-4d0b-8fbc-df6bf3108412-kube-api-access-46m8x\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.565586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.565522 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.565586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.565556 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.566500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.566466 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.569017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.568990 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.624247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.624217 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46m8x\" (UniqueName: \"kubernetes.io/projected/75439837-68b1-4d0b-8fbc-df6bf3108412-kube-api-access-46m8x\") pod \"maas-api-revoke-sub-e2c664cc-969jn\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:36.915622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:36.915530 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:37.007748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:37.007641 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:37.008928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:37.008901 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:37.009646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:37.009625 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 22:43:37.389842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:37.382975 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn"] Nov 03 22:43:37.430536 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:37.430504 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:37.430676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:37.430563 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:38.188268 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:38.188163 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" event={"ID":"75439837-68b1-4d0b-8fbc-df6bf3108412","Type":"ContainerStarted","Data":"1163c9eedd7b41a426cb0c9b785e6074857450c821b293a00f51a3c1c9efb56e"} Nov 03 22:43:38.188756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:38.188301 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" event={"ID":"75439837-68b1-4d0b-8fbc-df6bf3108412","Type":"ContainerStarted","Data":"c13b649c9da9d744d67c555eca5f9d37a8876a2b52dcd01304135ecf029dc589"} Nov 03 22:43:38.431343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:38.431303 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:38.431553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:38.431367 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:39.197988 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.197956 5272 generic.go:356] "Generic (PLEG): container finished" podID="75439837-68b1-4d0b-8fbc-df6bf3108412" containerID="1163c9eedd7b41a426cb0c9b785e6074857450c821b293a00f51a3c1c9efb56e" exitCode=0 Nov 03 22:43:39.198396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.198039 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" event={"ID":"75439837-68b1-4d0b-8fbc-df6bf3108412","Type":"ContainerDied","Data":"1163c9eedd7b41a426cb0c9b785e6074857450c821b293a00f51a3c1c9efb56e"} Nov 03 22:43:39.430901 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.430863 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:39.431093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.430925 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:39.573580 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.573531 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh"] Nov 03 22:43:39.579624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.579596 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623036 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-credential-socket\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623112 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-token\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623132 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqxh\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-kube-api-access-2wqxh\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623161 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-socket\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623176 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-envoy\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623205 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-data\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623543 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623366 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istiod-ca-cert\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623543 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623422 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-certs\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.623543 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.623437 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-podinfo\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.629403 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.629369 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bb95c832-openshift-default-dockercfg-6g49k\"" Nov 03 22:43:39.636096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.636064 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh"] Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789084 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-certs\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789128 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-podinfo\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789187 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-credential-socket\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789252 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-token\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789276 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqxh\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-kube-api-access-2wqxh\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789313 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-socket\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789338 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-envoy\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789379 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-data\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.789609 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istiod-ca-cert\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.790546 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istiod-ca-cert\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.800829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.790896 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-certs\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.822376 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.813444 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-credential-socket\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.867878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.865599 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-data\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.867878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.865673 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-socket\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.903841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.884279 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-podinfo\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:39.933836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.932488 5272 scope.go:122] "RemoveContainer" containerID="8e41df334faf3a0a376debee1cc19e711416c711134ddf93e927be0099217737" Nov 03 22:43:39.940825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:39.936253 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-envoy\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:40.030826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.016661 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqxh\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-kube-api-access-2wqxh\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:40.037928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.031464 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-token\") pod \"e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:40.111849 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.105919 5272 scope.go:122] "RemoveContainer" containerID="183ba58b7ffde9415223d38cbb7ca86bf3d74b1b0775ee3dfd8560def88b30db" Nov 03 22:43:40.201829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.196795 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:40.202405 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.201948 5272 scope.go:122] "RemoveContainer" containerID="7810407d7ce6fa2464fefa8aec24d6f9ae7351375f5c837abddc8273f7bd4e9f" Nov 03 22:43:40.421019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.419640 5272 scope.go:122] "RemoveContainer" containerID="97eacd6994a02ac5df88df2e97faa92a169967e929ea00587214cea5099be4de" Nov 03 22:43:40.430694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.430659 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:40.430868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.430724 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:40.487951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.487896 5272 scope.go:122] "RemoveContainer" containerID="90c1d0737905d6b655b7191548d008881f766c77a2218cc4e7e331b22be8f572" Nov 03 22:43:40.517082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.517054 5272 scope.go:122] "RemoveContainer" containerID="5310a7bd08d4988284c3c2be0d4f0a3765d9051bd8bcc4b6e0acbcd3d43c394d" Nov 03 22:43:40.534083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.533888 5272 scope.go:122] "RemoveContainer" containerID="858c97fe94ad34cbb589db03361843b7c29dcc400aabdcb5b03a05b908dbb19b" Nov 03 22:43:40.573874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.573843 5272 scope.go:122] "RemoveContainer" containerID="e3637455cb7b94d1386f1112888e1dc334d3c815682473f6310ce6dabf7d2c17" Nov 03 22:43:40.989139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:40.989111 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh"] Nov 03 22:43:40.989716 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:40.989688 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33bd25b_b162_47bd_aae3_e523f3ee31a7.slice/crio-4e1b68a5496222b2051b8c844b83a9a4c5150cf29098b5ed235ef1cbe5894a25 WatchSource:0}: Error finding container 4e1b68a5496222b2051b8c844b83a9a4c5150cf29098b5ed235ef1cbe5894a25: Status 404 returned error can't find the container with id 4e1b68a5496222b2051b8c844b83a9a4c5150cf29098b5ed235ef1cbe5894a25 Nov 03 22:43:41.015158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.015112 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:41.015281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.015224 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:41.015346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.015281 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:43:41.205491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.205467 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:41.321509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.321078 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-xks-ca\") pod \"75439837-68b1-4d0b-8fbc-df6bf3108412\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " Nov 03 22:43:41.321509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.321274 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75439837-68b1-4d0b-8fbc-df6bf3108412-kube-api-access-46m8x\" (UniqueName: \"kubernetes.io/projected/75439837-68b1-4d0b-8fbc-df6bf3108412-kube-api-access-46m8x\") pod \"75439837-68b1-4d0b-8fbc-df6bf3108412\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " Nov 03 22:43:41.321509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.321320 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-openshift-ca\") pod \"75439837-68b1-4d0b-8fbc-df6bf3108412\" (UID: \"75439837-68b1-4d0b-8fbc-df6bf3108412\") " Nov 03 22:43:41.321891 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.321766 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-openshift-ca" pod "75439837-68b1-4d0b-8fbc-df6bf3108412" (UID: "75439837-68b1-4d0b-8fbc-df6bf3108412"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:41.323328 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.323297 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75439837-68b1-4d0b-8fbc-df6bf3108412-kube-api-access-46m8x" pod "75439837-68b1-4d0b-8fbc-df6bf3108412" (UID: "75439837-68b1-4d0b-8fbc-df6bf3108412"). InnerVolumeSpecName "kube-api-access-46m8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:41.323638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.323613 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-xks-ca" pod "75439837-68b1-4d0b-8fbc-df6bf3108412" (UID: "75439837-68b1-4d0b-8fbc-df6bf3108412"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:41.417964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.417930 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" Nov 03 22:43:41.417964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.417957 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn" event={"ID":"75439837-68b1-4d0b-8fbc-df6bf3108412","Type":"ContainerDied","Data":"c13b649c9da9d744d67c555eca5f9d37a8876a2b52dcd01304135ecf029dc589"} Nov 03 22:43:41.418167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.418003 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c13b649c9da9d744d67c555eca5f9d37a8876a2b52dcd01304135ecf029dc589" Nov 03 22:43:41.420089 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.420058 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" event={"ID":"c33bd25b-b162-47bd-aae3-e523f3ee31a7","Type":"ContainerStarted","Data":"1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f"} Nov 03 22:43:41.420240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.420112 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" event={"ID":"c33bd25b-b162-47bd-aae3-e523f3ee31a7","Type":"ContainerStarted","Data":"4e1b68a5496222b2051b8c844b83a9a4c5150cf29098b5ed235ef1cbe5894a25"} Nov 03 22:43:41.422867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.422845 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46m8x\" (UniqueName: \"kubernetes.io/projected/75439837-68b1-4d0b-8fbc-df6bf3108412-kube-api-access-46m8x\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:41.422946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.422869 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:41.422946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.422879 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75439837-68b1-4d0b-8fbc-df6bf3108412-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:41.431278 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.431243 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:41.431431 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.431302 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:41.542243 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:41.542188 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podStartSLOduration=2.5421735439999997 podStartE2EDuration="2.542173544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:41.53834816 +0000 UTC m=+1202.160656470" watchObservedRunningTime="2026-11-03 22:43:41.542173544 +0000 UTC m=+1202.164481852" Nov 03 22:43:42.197783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:42.197703 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:42.204706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:42.204684 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:42.431823 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:42.431179 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:42.431823 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:42.431267 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:42.463889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:42.453876 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:42.485648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:42.474079 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:43:43.430929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:43.430891 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:43:43.431143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:43.430949 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 22:43:44.434981 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:44.434945 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:44.435447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:44.435430 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:44.436230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:44.436207 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:43:47.918827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:47.899403 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df"] Nov 03 22:43:47.918827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:47.900423 5272 cpu_manager.go:414] "containerMap: removing container" podUID="75439837-68b1-4d0b-8fbc-df6bf3108412" containerName="revoke-keys" Nov 03 22:43:47.918827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:47.900449 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75439837-68b1-4d0b-8fbc-df6bf3108412" containerName="revoke-keys" Nov 03 22:43:47.918827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:47.900659 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="75439837-68b1-4d0b-8fbc-df6bf3108412" containerName="revoke-keys" Nov 03 22:43:48.235892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.200051 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.236198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.202658 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 03 22:43:48.267041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.266982 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.288834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.281094 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded" Nov 03 22:43:48.384520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.344988 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.396836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.376094 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.500887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.498112 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df"] Nov 03 22:43:48.500887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.499354 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.597158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.597119 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:48.597639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.597612 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:43:48.597999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.597979 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:43:48.598246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.598228 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:43:48.640885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.640844 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfr8r\" (UniqueName: \"kubernetes.io/projected/8e4a8d65-e57d-42e7-a447-3ae0504e4328-kube-api-access-wfr8r\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.641571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.641544 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.641791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.641772 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.642367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.641212 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.683302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.683247 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.747825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.747018 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfr8r\" (UniqueName: \"kubernetes.io/projected/8e4a8d65-e57d-42e7-a447-3ae0504e4328-kube-api-access-wfr8r\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.747825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.747374 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.747825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.747462 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.751871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.749129 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.754826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.754782 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.814890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.814853 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfr8r\" (UniqueName: \"kubernetes.io/projected/8e4a8d65-e57d-42e7-a447-3ae0504e4328-kube-api-access-wfr8r\") pod \"maas-api-revoke-sub-b3f94c37-sw2df\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.825977 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.825930 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.838921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.838880 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.844074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.844039 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:48.926721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:48.925900 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.277956 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.277908 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 03 22:43:49.443828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.304382 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:43:49.465407 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.324895 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.466771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.325968 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:43:49.484934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.484900 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:43:49.505420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.390977 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.506077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.391995 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.573866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.338965 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:43:49.606855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.591052 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:43:49.606855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.591112 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:43:49.880331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.880211 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:49.882654 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.882619 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:43:49.883893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:49.883863 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.103341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.060353 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:43:50.103341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.060428 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:43:50.103341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.060468 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:43:50.156477 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.156379 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 22:43:50.158130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.158095 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:43:50.184063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.184026 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:43:50.184369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.184349 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:43:50.292100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.292051 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.352926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.352878 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:50.353250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.353200 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.493959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.481899 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.537653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.494591 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.562931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.517483 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 22:43:50.567922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.517619 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 03 22:43:50.642836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.628788 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:43:50.899827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.883009 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.900759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.900712 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:50.901031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.900988 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:43:50.901364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.901344 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:43:50.987759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:50.987713 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:51.178832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.129626 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:43:51.756845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.727547 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df"] Nov 03 22:43:51.757302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.161879 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:51.907591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.309081 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.038885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.501063 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.039077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.038955 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.039077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.715910 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 03 22:43:52.058036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.725073 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.092193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.907742 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.116726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.908140 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 03 22:43:52.116726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.908679 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:43:52.216318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.216207 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:43:52.301874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:51.157894 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.616138 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.616100 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:52.616434 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.616402 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:52.936888 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:43:52.920976 5272 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.073s" Nov 03 22:43:52.936888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.921049 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" event={"ID":"8e4a8d65-e57d-42e7-a447-3ae0504e4328","Type":"ContainerStarted","Data":"3b7033b9a8832ec63e8307ec28178cfe74f6d4591edb72eed4b4b60c927151e5"} Nov 03 22:43:52.963174 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.963150 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:43:52.982889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.965011 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:43:52.992254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:52.992219 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:43:53.004912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.004869 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.039889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.039296 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.076215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.076187 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:43:53.085899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.080604 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:43:53.107224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.106000 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.130826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.126367 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.130826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.128871 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:53.166912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.166866 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:43:53.184398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.184368 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:43:53.260554 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.259739 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:43:53.291828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.287299 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:43:53.301336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.301304 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:43:53.346379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.346273 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:43:53.527304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:53.527207 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" event={"ID":"8e4a8d65-e57d-42e7-a447-3ae0504e4328","Type":"ContainerStarted","Data":"f8c515ddcfb81c20fb025728f88fac8076e2bc1d89b92d73f88744d3d43510d3"} Nov 03 22:43:54.357136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:54.357084 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" podStartSLOduration=7.357071864 podStartE2EDuration="7.357071864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:54.352591889 +0000 UTC m=+1214.974900209" watchObservedRunningTime="2026-11-03 22:43:54.357071864 +0000 UTC m=+1214.979380224" Nov 03 22:43:54.509722 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:43:54.509680 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:43:54.537449 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:54.537413 5272 generic.go:356] "Generic (PLEG): container finished" podID="8e4a8d65-e57d-42e7-a447-3ae0504e4328" containerID="f8c515ddcfb81c20fb025728f88fac8076e2bc1d89b92d73f88744d3d43510d3" exitCode=0 Nov 03 22:43:54.537627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:54.537493 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" event={"ID":"8e4a8d65-e57d-42e7-a447-3ae0504e4328","Type":"ContainerDied","Data":"f8c515ddcfb81c20fb025728f88fac8076e2bc1d89b92d73f88744d3d43510d3"} Nov 03 22:43:55.722407 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.722381 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:55.822286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.822204 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk"] Nov 03 22:43:55.822863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.822848 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8e4a8d65-e57d-42e7-a447-3ae0504e4328" containerName="revoke-keys" Nov 03 22:43:55.822924 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.822865 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4a8d65-e57d-42e7-a447-3ae0504e4328" containerName="revoke-keys" Nov 03 22:43:55.822993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.822984 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4a8d65-e57d-42e7-a447-3ae0504e4328" containerName="revoke-keys" Nov 03 22:43:55.826659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.826639 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:55.849494 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.849461 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-xks-ca\") pod \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " Nov 03 22:43:55.849647 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.849631 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4a8d65-e57d-42e7-a447-3ae0504e4328-kube-api-access-wfr8r\" (UniqueName: \"kubernetes.io/projected/8e4a8d65-e57d-42e7-a447-3ae0504e4328-kube-api-access-wfr8r\") pod \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " Nov 03 22:43:55.849749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.849736 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-openshift-ca\") pod \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\" (UID: \"8e4a8d65-e57d-42e7-a447-3ae0504e4328\") " Nov 03 22:43:55.850115 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.850089 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-openshift-ca" pod "8e4a8d65-e57d-42e7-a447-3ae0504e4328" (UID: "8e4a8d65-e57d-42e7-a447-3ae0504e4328"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:55.850338 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.850319 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:55.851487 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.851465 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-xks-ca" pod "8e4a8d65-e57d-42e7-a447-3ae0504e4328" (UID: "8e4a8d65-e57d-42e7-a447-3ae0504e4328"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:55.851659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.851640 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4a8d65-e57d-42e7-a447-3ae0504e4328-kube-api-access-wfr8r" pod "8e4a8d65-e57d-42e7-a447-3ae0504e4328" (UID: "8e4a8d65-e57d-42e7-a447-3ae0504e4328"). InnerVolumeSpecName "kube-api-access-wfr8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:55.892307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.892265 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk"] Nov 03 22:43:55.951504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.951469 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:55.951671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.951512 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:55.951671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.951583 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp8ms\" (UniqueName: \"kubernetes.io/projected/c33f928f-81a5-46e7-a636-2d210a75170f-kube-api-access-kp8ms\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:55.951773 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.951726 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4a8d65-e57d-42e7-a447-3ae0504e4328-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:55.951773 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:55.951739 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfr8r\" (UniqueName: \"kubernetes.io/projected/8e4a8d65-e57d-42e7-a447-3ae0504e4328-kube-api-access-wfr8r\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:56.053395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.053354 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:56.053616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.053405 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:56.053616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.053447 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp8ms\" (UniqueName: \"kubernetes.io/projected/c33f928f-81a5-46e7-a636-2d210a75170f-kube-api-access-kp8ms\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:56.054136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.054112 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:56.056000 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.055970 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:56.111057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.110990 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp8ms\" (UniqueName: \"kubernetes.io/projected/c33f928f-81a5-46e7-a636-2d210a75170f-kube-api-access-kp8ms\") pod \"maas-api-revoke-sub-cd0440c5-f5sbk\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:56.140704 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.140679 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:56.522381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.522300 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk"] Nov 03 22:43:56.526027 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:43:56.525998 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33f928f_81a5_46e7_a636_2d210a75170f.slice/crio-59804f92b2f3a0ddd044abecaaea4e62de98ee35f5f04770bd296b79873b1d60 WatchSource:0}: Error finding container 59804f92b2f3a0ddd044abecaaea4e62de98ee35f5f04770bd296b79873b1d60: Status 404 returned error can't find the container with id 59804f92b2f3a0ddd044abecaaea4e62de98ee35f5f04770bd296b79873b1d60 Nov 03 22:43:56.557380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.557346 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" event={"ID":"c33f928f-81a5-46e7-a636-2d210a75170f","Type":"ContainerStarted","Data":"59804f92b2f3a0ddd044abecaaea4e62de98ee35f5f04770bd296b79873b1d60"} Nov 03 22:43:56.558747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.558729 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" Nov 03 22:43:56.558825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.558752 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df" event={"ID":"8e4a8d65-e57d-42e7-a447-3ae0504e4328","Type":"ContainerDied","Data":"3b7033b9a8832ec63e8307ec28178cfe74f6d4591edb72eed4b4b60c927151e5"} Nov 03 22:43:56.558868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:56.558830 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7033b9a8832ec63e8307ec28178cfe74f6d4591edb72eed4b4b60c927151e5" Nov 03 22:43:57.568743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:57.568704 5272 generic.go:356] "Generic (PLEG): container finished" podID="c33f928f-81a5-46e7-a636-2d210a75170f" containerID="f6dc0c4eae4ff606d2cfdd7726d5e6929cb8d37d5b84fe13ba7777f748af4676" exitCode=0 Nov 03 22:43:57.569263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:57.568788 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" event={"ID":"c33f928f-81a5-46e7-a636-2d210a75170f","Type":"ContainerDied","Data":"f6dc0c4eae4ff606d2cfdd7726d5e6929cb8d37d5b84fe13ba7777f748af4676"} Nov 03 22:43:58.748546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.748521 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:58.895256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.895153 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-openshift-ca\") pod \"c33f928f-81a5-46e7-a636-2d210a75170f\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " Nov 03 22:43:58.895256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.895256 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33f928f-81a5-46e7-a636-2d210a75170f-kube-api-access-kp8ms\" (UniqueName: \"kubernetes.io/projected/c33f928f-81a5-46e7-a636-2d210a75170f-kube-api-access-kp8ms\") pod \"c33f928f-81a5-46e7-a636-2d210a75170f\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " Nov 03 22:43:58.895553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.895533 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-xks-ca\") pod \"c33f928f-81a5-46e7-a636-2d210a75170f\" (UID: \"c33f928f-81a5-46e7-a636-2d210a75170f\") " Nov 03 22:43:58.895668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.895638 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-openshift-ca" pod "c33f928f-81a5-46e7-a636-2d210a75170f" (UID: "c33f928f-81a5-46e7-a636-2d210a75170f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:58.896326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.896297 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:58.897567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.897543 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33f928f-81a5-46e7-a636-2d210a75170f-kube-api-access-kp8ms" pod "c33f928f-81a5-46e7-a636-2d210a75170f" (UID: "c33f928f-81a5-46e7-a636-2d210a75170f"). InnerVolumeSpecName "kube-api-access-kp8ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:58.898059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.898038 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-xks-ca" pod "c33f928f-81a5-46e7-a636-2d210a75170f" (UID: "c33f928f-81a5-46e7-a636-2d210a75170f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:58.998315 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.998273 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp8ms\" (UniqueName: \"kubernetes.io/projected/c33f928f-81a5-46e7-a636-2d210a75170f-kube-api-access-kp8ms\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:58.998315 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:58.998306 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33f928f-81a5-46e7-a636-2d210a75170f-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:59.311781 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.311748 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k"] Nov 03 22:43:59.312433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.312416 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c33f928f-81a5-46e7-a636-2d210a75170f" containerName="revoke-keys" Nov 03 22:43:59.312493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.312434 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33f928f-81a5-46e7-a636-2d210a75170f" containerName="revoke-keys" Nov 03 22:43:59.312580 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.312570 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33f928f-81a5-46e7-a636-2d210a75170f" containerName="revoke-keys" Nov 03 22:43:59.316378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.316354 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.369785 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.369753 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k"] Nov 03 22:43:59.404199 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.404165 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.404199 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.404207 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6pn\" (UniqueName: \"kubernetes.io/projected/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-kube-api-access-6l6pn\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.404554 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.404530 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.505584 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.505543 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.505721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.505654 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.505721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.505679 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6pn\" (UniqueName: \"kubernetes.io/projected/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-kube-api-access-6l6pn\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.506343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.506316 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.507988 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.507965 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.563309 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.563233 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6pn\" (UniqueName: \"kubernetes.io/projected/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-kube-api-access-6l6pn\") pod \"maas-api-revoke-sub-e0ea7f22-v9b9k\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:43:59.587230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.587196 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" Nov 03 22:43:59.587230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.587226 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk" event={"ID":"c33f928f-81a5-46e7-a636-2d210a75170f","Type":"ContainerDied","Data":"59804f92b2f3a0ddd044abecaaea4e62de98ee35f5f04770bd296b79873b1d60"} Nov 03 22:43:59.587439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.587278 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59804f92b2f3a0ddd044abecaaea4e62de98ee35f5f04770bd296b79873b1d60" Nov 03 22:43:59.632125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:43:59.632087 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:44:00.480796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.480763 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2"] Nov 03 22:44:00.491786 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.491757 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.492196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.492173 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k"] Nov 03 22:44:00.502496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.502470 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2"] Nov 03 22:44:00.515297 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515262 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f08872-278e-4f08-9a4d-11e1e223321a-istiod-ca-cert\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515303 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-certs\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515412 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-data\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515449 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-envoy\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515509 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f08872-278e-4f08-9a4d-11e1e223321a-istio-podinfo\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515541 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-istio-token\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515632 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-socket\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515791 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-credential-socket\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.515875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.515847 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqb2r\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-kube-api-access-qqb2r\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.540829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.540784 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c082ab23-openshift-default-dockercfg-zvj2p\"" Nov 03 22:44:00.598281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.598238 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" event={"ID":"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95","Type":"ContainerStarted","Data":"40ed5198858db7740cb4f9b2ac230f62a1a040c88fbb1d80d081c5573ef73070"} Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.636703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-credential-socket\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.636773 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqb2r\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-kube-api-access-qqb2r\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637243 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-credential-socket\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637405 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f08872-278e-4f08-9a4d-11e1e223321a-istiod-ca-cert\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637450 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-certs\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637522 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-data\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637566 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-envoy\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637635 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f08872-278e-4f08-9a4d-11e1e223321a-istio-podinfo\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637665 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-istio-token\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.637727 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-socket\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.638267 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-socket\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.639205 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f08872-278e-4f08-9a4d-11e1e223321a-istiod-ca-cert\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.639468 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-certs\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.646877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.639682 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-data\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.662835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.662729 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f08872-278e-4f08-9a4d-11e1e223321a-istio-podinfo\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.665840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.665354 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-envoy\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.732828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.730851 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqb2r\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-kube-api-access-qqb2r\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.732828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.732514 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-istio-token\") pod \"e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:00.838876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:00.812640 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:01.267239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.267207 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2"] Nov 03 22:44:01.271405 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:44:01.271369 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f08872_278e_4f08_9a4d_11e1e223321a.slice/crio-6d8f4d575b0cbbbe9f800d7915f06881beaa00389812baad9ef6ab7aeea1970a WatchSource:0}: Error finding container 6d8f4d575b0cbbbe9f800d7915f06881beaa00389812baad9ef6ab7aeea1970a: Status 404 returned error can't find the container with id 6d8f4d575b0cbbbe9f800d7915f06881beaa00389812baad9ef6ab7aeea1970a Nov 03 22:44:01.274257 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.274218 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:01.274373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.274310 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:01.274458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.274381 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:01.730872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.730074 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" event={"ID":"77f08872-278e-4f08-9a4d-11e1e223321a","Type":"ContainerStarted","Data":"7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60"} Nov 03 22:44:01.730872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.730187 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" event={"ID":"77f08872-278e-4f08-9a4d-11e1e223321a","Type":"ContainerStarted","Data":"6d8f4d575b0cbbbe9f800d7915f06881beaa00389812baad9ef6ab7aeea1970a"} Nov 03 22:44:01.774880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.774593 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" event={"ID":"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95","Type":"ContainerStarted","Data":"c315def20b5e272ce495b4c2c138e24b36db56083e8c28afb2b3069a7e92a7f9"} Nov 03 22:44:01.815032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.814943 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:01.857936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:01.857874 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podStartSLOduration=2.857855526 podStartE2EDuration="2.857855526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:01.853500343 +0000 UTC m=+1222.475808651" watchObservedRunningTime="2026-11-03 22:44:01.857855526 +0000 UTC m=+1222.480163836" Nov 03 22:44:02.786130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.786092 5272 generic.go:356] "Generic (PLEG): container finished" podID="f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" containerID="c315def20b5e272ce495b4c2c138e24b36db56083e8c28afb2b3069a7e92a7f9" exitCode=0 Nov 03 22:44:02.786577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.786173 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" event={"ID":"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95","Type":"ContainerDied","Data":"c315def20b5e272ce495b4c2c138e24b36db56083e8c28afb2b3069a7e92a7f9"} Nov 03 22:44:02.820089 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.820060 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:02.953699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.953674 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:44:02.959038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.959014 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-kube-api-access-6l6pn\" (UniqueName: \"kubernetes.io/projected/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-kube-api-access-6l6pn\") pod \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " Nov 03 22:44:02.959162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.959109 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-xks-ca\") pod \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " Nov 03 22:44:02.959307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.959292 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-openshift-ca\") pod \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\" (UID: \"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95\") " Nov 03 22:44:02.959637 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.959604 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-openshift-ca" pod "f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" (UID: "f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:02.960051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.960032 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:02.961087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.961064 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-xks-ca" pod "f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" (UID: "f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:02.961151 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:02.961115 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-kube-api-access-6l6pn" pod "f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" (UID: "f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95"). InnerVolumeSpecName "kube-api-access-6l6pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:03.061422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:03.061283 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:03.061422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:03.061356 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6pn\" (UniqueName: \"kubernetes.io/projected/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95-kube-api-access-6l6pn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:03.810733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:03.810609 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" event={"ID":"f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95","Type":"ContainerDied","Data":"40ed5198858db7740cb4f9b2ac230f62a1a040c88fbb1d80d081c5573ef73070"} Nov 03 22:44:03.810733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:03.810675 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ed5198858db7740cb4f9b2ac230f62a1a040c88fbb1d80d081c5573ef73070" Nov 03 22:44:03.810733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:03.810697 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:03.811476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:03.811450 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k" Nov 03 22:44:03.832834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:03.828137 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:44:06.852342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.852291 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg"] Nov 03 22:44:06.853347 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.853324 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" containerName="revoke-keys" Nov 03 22:44:06.853419 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.853350 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" containerName="revoke-keys" Nov 03 22:44:06.853510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.853499 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" containerName="revoke-keys" Nov 03 22:44:06.858355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.858327 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg"] Nov 03 22:44:06.858502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.858423 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:06.907167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.907138 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:06.953001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.952964 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:06.953167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.953025 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thpt\" (UniqueName: \"kubernetes.io/projected/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-kube-api-access-9thpt\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:06.953167 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.953137 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:06.953433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.953410 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:44:06.953568 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.953536 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:44:06.953643 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.953569 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:44:06.958383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.958358 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795"] Nov 03 22:44:06.968107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.966163 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:06.988473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:06.988420 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795"] Nov 03 22:44:07.018159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.018122 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-027a6fbe\"" Nov 03 22:44:07.018608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.018581 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-027a6fbe-dockercfg-59dkt\"" Nov 03 22:44:07.235871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.231550 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thpt\" (UniqueName: \"kubernetes.io/projected/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-kube-api-access-9thpt\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:07.235871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.231626 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:07.235871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.231700 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:07.235871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.231747 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27vf\" (UniqueName: \"kubernetes.io/projected/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-kube-api-access-r27vf\") pod \"payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:07.235871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.232033 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:07.235871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.232859 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:07.295503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.295453 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thpt\" (UniqueName: \"kubernetes.io/projected/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-kube-api-access-9thpt\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:07.330304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.330262 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0750d0d7-r7hcg\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:07.335369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.335327 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:07.337360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.337333 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27vf\" (UniqueName: \"kubernetes.io/projected/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-kube-api-access-r27vf\") pod \"payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:07.338065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.337251 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:07.338947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.338925 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn"] Nov 03 22:44:07.360671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.360637 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.363022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.362998 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn"] Nov 03 22:44:07.397473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.397441 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27vf\" (UniqueName: \"kubernetes.io/projected/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-kube-api-access-r27vf\") pod \"payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:07.439530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.439484 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcbd3c5-4447-4a79-b5f4-7140966c958d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.439720 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.439621 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhh2\" (UniqueName: \"kubernetes.io/projected/1bcbd3c5-4447-4a79-b5f4-7140966c958d-kube-api-access-8hhh2\") pod \"payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.473982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.473942 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:07.547846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.541631 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcbd3c5-4447-4a79-b5f4-7140966c958d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.547846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.542563 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcbd3c5-4447-4a79-b5f4-7140966c958d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.547846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.542706 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhh2\" (UniqueName: \"kubernetes.io/projected/1bcbd3c5-4447-4a79-b5f4-7140966c958d-kube-api-access-8hhh2\") pod \"payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.586516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.586478 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:07.602241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.602212 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhh2\" (UniqueName: \"kubernetes.io/projected/1bcbd3c5-4447-4a79-b5f4-7140966c958d-kube-api-access-8hhh2\") pod \"payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.702460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.702415 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:07.964094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:07.964067 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg"] Nov 03 22:44:08.068647 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.068616 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795"] Nov 03 22:44:08.069812 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:44:08.069776 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0970e14_cd44_44db_b63b_7d5c6fd7d3d5.slice/crio-fce988741f4c898381b08771bb32ccc39cbdd5c2eb5f0fe46ad9786b1fed36e7 WatchSource:0}: Error finding container fce988741f4c898381b08771bb32ccc39cbdd5c2eb5f0fe46ad9786b1fed36e7: Status 404 returned error can't find the container with id fce988741f4c898381b08771bb32ccc39cbdd5c2eb5f0fe46ad9786b1fed36e7 Nov 03 22:44:08.091134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.091110 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn"] Nov 03 22:44:08.109998 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:44:08.109965 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcbd3c5_4447_4a79_b5f4_7140966c958d.slice/crio-76a7c896dae4845fd0ece7b7b5ee00ddf3be8c4a543116e1a581e6513ade7fc5 WatchSource:0}: Error finding container 76a7c896dae4845fd0ece7b7b5ee00ddf3be8c4a543116e1a581e6513ade7fc5: Status 404 returned error can't find the container with id 76a7c896dae4845fd0ece7b7b5ee00ddf3be8c4a543116e1a581e6513ade7fc5 Nov 03 22:44:08.971908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.971867 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" event={"ID":"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46","Type":"ContainerStarted","Data":"7208a7d0bf36728c5e93c2d3ce4a4167a4be39a1803d2cab5d4f31f29c202905"} Nov 03 22:44:08.972352 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.971935 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" event={"ID":"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46","Type":"ContainerStarted","Data":"ec41154fc8c511fe39329848e5891af6a0ad49dcb4e6d500b9a92ed518c3e792"} Nov 03 22:44:08.973710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.973676 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" event={"ID":"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5","Type":"ContainerStarted","Data":"887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1"} Nov 03 22:44:08.973846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.973731 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" event={"ID":"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5","Type":"ContainerStarted","Data":"fce988741f4c898381b08771bb32ccc39cbdd5c2eb5f0fe46ad9786b1fed36e7"} Nov 03 22:44:08.973846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.973759 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:08.975694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.975668 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" event={"ID":"1bcbd3c5-4447-4a79-b5f4-7140966c958d","Type":"ContainerStarted","Data":"71254ed5fd0d80a1d7f361f844a11a879f33da7df38e86a369cdfa90958ca302"} Nov 03 22:44:08.975819 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.975714 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:08.975819 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:08.975733 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" event={"ID":"1bcbd3c5-4447-4a79-b5f4-7140966c958d","Type":"ContainerStarted","Data":"76a7c896dae4845fd0ece7b7b5ee00ddf3be8c4a543116e1a581e6513ade7fc5"} Nov 03 22:44:09.083194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:09.083142 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" podStartSLOduration=3.083126457 podStartE2EDuration="3.083126457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:09.080616253 +0000 UTC m=+1229.702924560" watchObservedRunningTime="2026-11-03 22:44:09.083126457 +0000 UTC m=+1229.705434765" Nov 03 22:44:09.191658 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:09.191602 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" podStartSLOduration=3.191585651 podStartE2EDuration="3.191585651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:09.189008837 +0000 UTC m=+1229.811317146" watchObservedRunningTime="2026-11-03 22:44:09.191585651 +0000 UTC m=+1229.813893958" Nov 03 22:44:09.303978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:09.303924 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" podStartSLOduration=2.303910391 podStartE2EDuration="2.303910391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:09.299481568 +0000 UTC m=+1229.921789877" watchObservedRunningTime="2026-11-03 22:44:09.303910391 +0000 UTC m=+1229.926218699" Nov 03 22:44:09.985351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:09.985317 5272 generic.go:356] "Generic (PLEG): container finished" podID="cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" containerID="7208a7d0bf36728c5e93c2d3ce4a4167a4be39a1803d2cab5d4f31f29c202905" exitCode=0 Nov 03 22:44:09.985747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:09.985376 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" event={"ID":"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46","Type":"ContainerDied","Data":"7208a7d0bf36728c5e93c2d3ce4a4167a4be39a1803d2cab5d4f31f29c202905"} Nov 03 22:44:10.227180 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.227104 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl"] Nov 03 22:44:10.231746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.231728 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.275816 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.275781 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9e954-4292-440d-866f-787832480959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.275946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.275898 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9hq\" (UniqueName: \"kubernetes.io/projected/b5d9e954-4292-440d-866f-787832480959-kube-api-access-xl9hq\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.276014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.275968 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9e954-4292-440d-866f-787832480959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.285748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.285715 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl"] Nov 03 22:44:10.377114 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.377074 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9e954-4292-440d-866f-787832480959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.377316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.377228 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9hq\" (UniqueName: \"kubernetes.io/projected/b5d9e954-4292-440d-866f-787832480959-kube-api-access-xl9hq\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.377395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.377322 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9e954-4292-440d-866f-787832480959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.378096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.378064 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9e954-4292-440d-866f-787832480959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.379688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.379667 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9e954-4292-440d-866f-787832480959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.434725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.434695 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9hq\" (UniqueName: \"kubernetes.io/projected/b5d9e954-4292-440d-866f-787832480959-kube-api-access-xl9hq\") pod \"maas-api-revoke-sub-60c2fd7b-skfdl\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.545934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.545900 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:10.927632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.927556 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl"] Nov 03 22:44:10.994868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:10.994837 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" event={"ID":"b5d9e954-4292-440d-866f-787832480959","Type":"ContainerStarted","Data":"1761493b9589f4ae96a8c5eb1a30ebf9991a9af59a1885cb9533ffb27e74f8e1"} Nov 03 22:44:11.124217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.124190 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:11.189663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.189590 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-openshift-ca\") pod \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " Nov 03 22:44:11.189663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.189625 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-kube-api-access-9thpt\" (UniqueName: \"kubernetes.io/projected/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-kube-api-access-9thpt\") pod \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " Nov 03 22:44:11.189863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.189691 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-xks-ca\") pod \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\" (UID: \"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46\") " Nov 03 22:44:11.190008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.189980 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-openshift-ca" pod "cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" (UID: "cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:11.190244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.190226 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:11.192001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.191974 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-xks-ca" pod "cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" (UID: "cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:11.192096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.192039 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-kube-api-access-9thpt" pod "cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" (UID: "cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46"). InnerVolumeSpecName "kube-api-access-9thpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:11.291482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.291441 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:11.291482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:11.291477 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thpt\" (UniqueName: \"kubernetes.io/projected/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46-kube-api-access-9thpt\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:12.004745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:12.004707 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" Nov 03 22:44:12.004745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:12.004729 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg" event={"ID":"cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46","Type":"ContainerDied","Data":"ec41154fc8c511fe39329848e5891af6a0ad49dcb4e6d500b9a92ed518c3e792"} Nov 03 22:44:12.005195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:12.004821 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec41154fc8c511fe39329848e5891af6a0ad49dcb4e6d500b9a92ed518c3e792" Nov 03 22:44:12.006750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:12.006720 5272 generic.go:356] "Generic (PLEG): container finished" podID="b5d9e954-4292-440d-866f-787832480959" containerID="331d9545a943efb32027dec03eb18315c7c8f0a7636431b372dcc54182c3fe37" exitCode=0 Nov 03 22:44:12.006974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:12.006775 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" event={"ID":"b5d9e954-4292-440d-866f-787832480959","Type":"ContainerDied","Data":"331d9545a943efb32027dec03eb18315c7c8f0a7636431b372dcc54182c3fe37"} Nov 03 22:44:13.185777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.185754 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:13.217240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.217210 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d9e954-4292-440d-866f-787832480959-kube-api-access-xl9hq\" (UniqueName: \"kubernetes.io/projected/b5d9e954-4292-440d-866f-787832480959-kube-api-access-xl9hq\") pod \"b5d9e954-4292-440d-866f-787832480959\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " Nov 03 22:44:13.217390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.217251 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d9e954-4292-440d-866f-787832480959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9e954-4292-440d-866f-787832480959-maas-api-xks-ca\") pod \"b5d9e954-4292-440d-866f-787832480959\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " Nov 03 22:44:13.217390 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.217308 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d9e954-4292-440d-866f-787832480959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9e954-4292-440d-866f-787832480959-maas-api-openshift-ca\") pod \"b5d9e954-4292-440d-866f-787832480959\" (UID: \"b5d9e954-4292-440d-866f-787832480959\") " Nov 03 22:44:13.217748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.217719 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d9e954-4292-440d-866f-787832480959-maas-api-openshift-ca" pod "b5d9e954-4292-440d-866f-787832480959" (UID: "b5d9e954-4292-440d-866f-787832480959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:13.218066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.218047 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d9e954-4292-440d-866f-787832480959-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.219343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.219315 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d9e954-4292-440d-866f-787832480959-maas-api-xks-ca" pod "b5d9e954-4292-440d-866f-787832480959" (UID: "b5d9e954-4292-440d-866f-787832480959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:13.219466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.219380 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d9e954-4292-440d-866f-787832480959-kube-api-access-xl9hq" pod "b5d9e954-4292-440d-866f-787832480959" (UID: "b5d9e954-4292-440d-866f-787832480959"). InnerVolumeSpecName "kube-api-access-xl9hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:13.319632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.319595 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9hq\" (UniqueName: \"kubernetes.io/projected/b5d9e954-4292-440d-866f-787832480959-kube-api-access-xl9hq\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.319632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:13.319635 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d9e954-4292-440d-866f-787832480959-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:14.026348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:14.026320 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" Nov 03 22:44:14.026526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:14.026351 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl" event={"ID":"b5d9e954-4292-440d-866f-787832480959","Type":"ContainerDied","Data":"1761493b9589f4ae96a8c5eb1a30ebf9991a9af59a1885cb9533ffb27e74f8e1"} Nov 03 22:44:14.026526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:14.026398 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1761493b9589f4ae96a8c5eb1a30ebf9991a9af59a1885cb9533ffb27e74f8e1" Nov 03 22:44:19.380107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.380067 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg"] Nov 03 22:44:19.404740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.404704 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b5d9e954-4292-440d-866f-787832480959" containerName="revoke-keys" Nov 03 22:44:19.410992 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.410951 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d9e954-4292-440d-866f-787832480959" containerName="revoke-keys" Nov 03 22:44:19.411491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.411423 5272 cpu_manager.go:414] "containerMap: removing container" podUID="cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" containerName="revoke-keys" Nov 03 22:44:19.411728 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.411712 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" containerName="revoke-keys" Nov 03 22:44:19.416819 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.416787 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d9e954-4292-440d-866f-787832480959" containerName="revoke-keys" Nov 03 22:44:19.416980 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.416968 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" containerName="revoke-keys" Nov 03 22:44:19.460206 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.460173 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck"] Nov 03 22:44:19.466959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.466925 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.542883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.542338 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.542883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.542504 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p296\" (UniqueName: \"kubernetes.io/projected/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-kube-api-access-2p296\") pod \"payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.611044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.611013 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c082ab23\"" Nov 03 22:44:19.612254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.612125 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c082ab23-dockercfg-8dwtr\"" Nov 03 22:44:19.659965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.647896 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p296\" (UniqueName: \"kubernetes.io/projected/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-kube-api-access-2p296\") pod \"payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.659965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.648336 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.659965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.649355 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.732071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.729642 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p296\" (UniqueName: \"kubernetes.io/projected/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-kube-api-access-2p296\") pod \"payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.767864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.753038 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.768190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.768156 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.768483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.768460 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.768608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.768593 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.768703 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.768690 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.768836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.768818 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.768992 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.768975 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.769214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.769198 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bws\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-kube-api-access-97bws\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.769355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.769338 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.773589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.773566 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr"] Nov 03 22:44:19.777687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.774485 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.795423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.795394 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg"] Nov 03 22:44:19.798074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.797951 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.819038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.819008 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck"] Nov 03 22:44:19.819186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.819060 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr"] Nov 03 22:44:19.819186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.819081 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j"] Nov 03 22:44:19.823785 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.823760 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:19.846046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.846010 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j"] Nov 03 22:44:19.871016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.870973 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.871191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871029 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a23076-aa94-4308-9c33-a59ecb05b90a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:19.871191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871180 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.871323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871218 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25qcs\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-kube-api-access-25qcs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.871323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871265 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dd63fa-31cb-488e-8232-b07a170bfc44-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.871435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871391 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.871496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871455 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.871554 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871500 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.871607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871578 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhb4\" (UniqueName: \"kubernetes.io/projected/c8a23076-aa94-4308-9c33-a59ecb05b90a-kube-api-access-9lhb4\") pod \"payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:19.871730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871709 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.871794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871746 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.871794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871770 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.871923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871832 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.871923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.871871 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.872151 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872123 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.872151 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872123 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.872466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872443 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.872633 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872619 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.872709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872670 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.872709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872693 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.872853 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872739 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.872853 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872825 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.872975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872870 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bws\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-kube-api-access-97bws\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.872975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.872927 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.873083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.873013 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.879222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.879187 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.881575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.881533 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.929759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.929687 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.931301 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.931281 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bws\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-kube-api-access-97bws\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:19.940402 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.940380 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:19.974286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974257 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.974388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974307 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a23076-aa94-4308-9c33-a59ecb05b90a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:19.974388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974349 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.974742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974722 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-data\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.974844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974769 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25qcs\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-kube-api-access-25qcs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.974899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974838 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dd63fa-31cb-488e-8232-b07a170bfc44-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.974960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974921 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.974960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.974953 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhb4\" (UniqueName: \"kubernetes.io/projected/c8a23076-aa94-4308-9c33-a59ecb05b90a-kube-api-access-9lhb4\") pod \"payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:19.975069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975035 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.975069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975064 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.975175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975086 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.975175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975123 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.975297 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975219 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a23076-aa94-4308-9c33-a59ecb05b90a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:19.975437 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975404 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-certs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.975949 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975899 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-credential-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.975949 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.975923 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-socket\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.976122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.976105 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dd63fa-31cb-488e-8232-b07a170bfc44-istiod-ca-cert\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.977160 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.977135 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-podinfo\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.977659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.977631 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-envoy\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:19.988910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.988884 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:44:19.989037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:19.988970 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:44:20.033379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.033346 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25qcs\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-kube-api-access-25qcs\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:20.034621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.034568 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-token\") pod \"e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:20.034621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.034587 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhb4\" (UniqueName: \"kubernetes.io/projected/c8a23076-aa94-4308-9c33-a59ecb05b90a-kube-api-access-9lhb4\") pod \"payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:20.110164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.110125 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:20.124064 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.124031 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:20.142371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.142291 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:20.357192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.356215 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg"] Nov 03 22:44:20.357192 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:44:20.357074 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb9bf3c_5a18_4cb5_8fca_9f4baa200965.slice/crio-943bb0d90701270c95cde4e7678c8c0cb79bcff33cad859b3f031fba6f1197e4 WatchSource:0}: Error finding container 943bb0d90701270c95cde4e7678c8c0cb79bcff33cad859b3f031fba6f1197e4: Status 404 returned error can't find the container with id 943bb0d90701270c95cde4e7678c8c0cb79bcff33cad859b3f031fba6f1197e4 Nov 03 22:44:20.516377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.516343 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck"] Nov 03 22:44:20.517545 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.517466 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:20.517666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.517641 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:20.517754 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.517731 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:20.561103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.561076 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j"] Nov 03 22:44:20.564511 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:44:20.564443 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a23076_aa94_4308_9c33_a59ecb05b90a.slice/crio-be824cad058ba686e3bbd995088ec404c7c6fee932d8b35a2994ea9f25079351 WatchSource:0}: Error finding container be824cad058ba686e3bbd995088ec404c7c6fee932d8b35a2994ea9f25079351: Status 404 returned error can't find the container with id be824cad058ba686e3bbd995088ec404c7c6fee932d8b35a2994ea9f25079351 Nov 03 22:44:20.565855 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:44:20.565794 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dd63fa_31cb_488e_8232_b07a170bfc44.slice/crio-6d74d321c927566572a19d963d3a173b5b6712cee28d7e5f747ecd491c89fdea WatchSource:0}: Error finding container 6d74d321c927566572a19d963d3a173b5b6712cee28d7e5f747ecd491c89fdea: Status 404 returned error can't find the container with id 6d74d321c927566572a19d963d3a173b5b6712cee28d7e5f747ecd491c89fdea Nov 03 22:44:20.567586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.567554 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr"] Nov 03 22:44:20.572130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.572089 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:20.572406 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.572378 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:20.572616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:20.572593 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:21.296742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.296704 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" event={"ID":"03dd63fa-31cb-488e-8232-b07a170bfc44","Type":"ContainerStarted","Data":"eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b"} Nov 03 22:44:21.296936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.296768 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" event={"ID":"03dd63fa-31cb-488e-8232-b07a170bfc44","Type":"ContainerStarted","Data":"6d74d321c927566572a19d963d3a173b5b6712cee28d7e5f747ecd491c89fdea"} Nov 03 22:44:21.298582 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.298555 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" event={"ID":"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965","Type":"ContainerStarted","Data":"d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a"} Nov 03 22:44:21.298742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.298599 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" event={"ID":"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965","Type":"ContainerStarted","Data":"943bb0d90701270c95cde4e7678c8c0cb79bcff33cad859b3f031fba6f1197e4"} Nov 03 22:44:21.298742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.298653 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:21.300513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.300485 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" event={"ID":"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3","Type":"ContainerStarted","Data":"3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8"} Nov 03 22:44:21.300653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.300541 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" event={"ID":"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3","Type":"ContainerStarted","Data":"f602e4616a9d0cb5025a970d7375bdcf7d5c6ba62c1c7f3b6af020a19e8acd10"} Nov 03 22:44:21.302025 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.302001 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" event={"ID":"c8a23076-aa94-4308-9c33-a59ecb05b90a","Type":"ContainerStarted","Data":"fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28"} Nov 03 22:44:21.302146 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.302036 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" event={"ID":"c8a23076-aa94-4308-9c33-a59ecb05b90a","Type":"ContainerStarted","Data":"be824cad058ba686e3bbd995088ec404c7c6fee932d8b35a2994ea9f25079351"} Nov 03 22:44:21.302146 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.302139 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:21.420766 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.419884 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podStartSLOduration=2.419860243 podStartE2EDuration="2.419860243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:21.410629483 +0000 UTC m=+1242.032937840" watchObservedRunningTime="2026-11-03 22:44:21.419860243 +0000 UTC m=+1242.042168552" Nov 03 22:44:21.524168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.524110 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" podStartSLOduration=2.5240954799999997 podStartE2EDuration="2.52409548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:21.523737383 +0000 UTC m=+1242.146045693" watchObservedRunningTime="2026-11-03 22:44:21.52409548 +0000 UTC m=+1242.146403787" Nov 03 22:44:21.767883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.767431 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podStartSLOduration=2.7674115219999997 podStartE2EDuration="2.767411522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:21.639707254 +0000 UTC m=+1242.262015563" watchObservedRunningTime="2026-11-03 22:44:21.767411522 +0000 UTC m=+1242.389719831" Nov 03 22:44:21.802829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:21.801165 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" podStartSLOduration=2.801141534 podStartE2EDuration="2.801141534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:21.766299078 +0000 UTC m=+1242.388607389" watchObservedRunningTime="2026-11-03 22:44:21.801141534 +0000 UTC m=+1242.423449844" Nov 03 22:44:22.113369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:22.111439 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:22.116652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:22.116464 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:22.116652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:22.116533 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:22.126822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:22.124142 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:22.126822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:22.125260 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:22.126822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:22.125308 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:23.110873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:23.110819 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:23.111290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:23.110888 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:23.126028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:23.125113 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:23.126028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:23.125175 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:24.111436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.111380 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:24.112029 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.111984 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:24.125083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.125052 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:24.125204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.125096 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:24.695828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.687561 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2"] Nov 03 22:44:24.721358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.712773 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:24.779829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.769304 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c6905007\"" Nov 03 22:44:24.834275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.831570 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c6905007-dockercfg-rm79b\"" Nov 03 22:44:24.866190 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:44:24.864819 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:44:24.915245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.908041 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d77b8c-8686-49b4-b35a-290dc6bafdfe-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:24.915245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.908335 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdnr\" (UniqueName: \"kubernetes.io/projected/49d77b8c-8686-49b4-b35a-290dc6bafdfe-kube-api-access-nhdnr\") pod \"payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:24.947885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:24.935389 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2"] Nov 03 22:44:25.177835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.009621 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdnr\" (UniqueName: \"kubernetes.io/projected/49d77b8c-8686-49b4-b35a-290dc6bafdfe-kube-api-access-nhdnr\") pod \"payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:25.177835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.009874 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d77b8c-8686-49b4-b35a-290dc6bafdfe-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:25.177835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.010977 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d77b8c-8686-49b4-b35a-290dc6bafdfe-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:25.205020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.204929 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:25.205290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.205258 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:25.222339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.222295 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:25.222621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.222590 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:25.283828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.230456 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdnr\" (UniqueName: \"kubernetes.io/projected/49d77b8c-8686-49b4-b35a-290dc6bafdfe-kube-api-access-nhdnr\") pod \"payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:25.396468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.396110 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:25.490947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.440519 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:44:25.587851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.573355 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 22:44:25.867258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.867211 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj"] Nov 03 22:44:25.913053 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.912995 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:25.985017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:25.943064 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:44:26.030065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.008044 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.040663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.040634 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj"] Nov 03 22:44:26.138830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.134876 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:26.158844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.139062 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:26.158844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.158049 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:26.158844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.158109 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:26.161792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.161758 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.162041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.162020 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9k7\" (UniqueName: \"kubernetes.io/projected/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-kube-api-access-7f9k7\") pod \"payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.270485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.267958 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.270485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.268005 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9k7\" (UniqueName: \"kubernetes.io/projected/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-kube-api-access-7f9k7\") pod \"payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.270485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.269712 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.429885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.407932 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:26.429885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.421501 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9k7\" (UniqueName: \"kubernetes.io/projected/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-kube-api-access-7f9k7\") pod \"payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.617063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.595491 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:44:26.693120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.691840 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:26.881317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.881153 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" event={"ID":"49d77b8c-8686-49b4-b35a-290dc6bafdfe","Type":"ContainerStarted","Data":"04845b7324ac863fe46cdf3c4170ac89f18f98f6287ade93f0dd9ea1ee1c343c"} Nov 03 22:44:26.895305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:26.895232 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2"] Nov 03 22:44:27.153092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.144902 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:27.205185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.196856 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:27.205185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.196960 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:27.273827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.273341 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:27.273827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.273404 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:27.374208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.368939 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:27.374208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.369000 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:27.422889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.421964 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:27.422889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.422537 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:44:27.586827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.582619 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:27.635651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.635609 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:27.947739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.946949 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:27.950829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.948346 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:44:27.950829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.950162 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:27.983832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:27.978908 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 22:44:28.034922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.022377 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.113852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.076138 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.145966 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.145897 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.222600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.149463 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:28.222600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.149516 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:28.222600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.149588 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:28.222600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.149612 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:28.222600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.150399 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.222600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.150722 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:44:28.222600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.166923 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.439946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.439894 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.475887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.465330 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:44:28.609828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.607614 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.667842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.616780 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:44:28.667842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.643005 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:28.667842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.643428 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:44:28.832898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.766465 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" event={"ID":"49d77b8c-8686-49b4-b35a-290dc6bafdfe","Type":"ContainerStarted","Data":"24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba"} Nov 03 22:44:28.832898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.766711 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:28.956938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.956891 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.289504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:28.964102 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.290040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.290016 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:44:29.290221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.290209 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:44:29.291596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.048322 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.318621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.193876 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:44:29.369046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.193916 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 03 22:44:29.369532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.265449 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:29.369716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.369685 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:29.369869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.183004 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.370206 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.048261 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.370567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.265494 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:29.370711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.370686 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:29.370843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.318901 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:44:29.371021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.371007 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:44:29.399515 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.399484 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:44:29.401546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.401521 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:44:29.452107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.446121 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.512928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.501331 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:44:29.512928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.504062 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:29.731394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:29.731257 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:30.299916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:30.290735 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:44:30.375996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:30.320423 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 03 22:44:30.394010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:30.329555 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:30.550454 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:30.549883 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:44:30.552925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:30.552873 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:30.744848 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:30.731758 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.076828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.073179 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:31.077161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.077107 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:31.127889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.115461 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.127889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.115921 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.157409 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.157366 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:31.157694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.157658 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:31.222059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.222017 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:31.225916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.225872 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:31.237835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.226437 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:31.238211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.238178 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:31.290427 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.248351 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.290427 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.248679 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 22:44:31.291587 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.291292 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.376969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.376918 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.423940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.423835 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-ggsmt" podUID="7f7a50d4-86b8-404f-99db-2b3be5d3c7cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.452049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.452005 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.510100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.510056 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 03 22:44:31.669827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.599654 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.669827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.600119 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 03 22:44:31.741713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.741610 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.795828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.779269 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:31.979863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:31.973077 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" podStartSLOduration=7.973055141 podStartE2EDuration="7.973055141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:29.079953455 +0000 UTC m=+1249.702261766" watchObservedRunningTime="2026-11-03 22:44:31.973055141 +0000 UTC m=+1252.595363452" Nov 03 22:44:32.047842 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:44:32.044784 5272 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.622s" Nov 03 22:44:32.047842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.044846 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj"] Nov 03 22:44:32.127827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.112916 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:32.127827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.112997 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:32.127827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.116896 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:32.127827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.120878 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 22:44:32.234071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.234025 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 22:44:32.272825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.267777 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 22:44:32.272825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.269698 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:32.420901 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.413240 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 03 22:44:32.420901 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.413642 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:32.464357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.464318 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:44:32.464579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.464382 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 22:44:32.465950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.465921 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:44:32.466105 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.465986 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" event={"ID":"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e","Type":"ContainerStarted","Data":"0a6f883daf56273186342f9c380e8d35ffbe950fb97ec5130587a09613907206"} Nov 03 22:44:32.467530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.467510 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 22:44:32.468505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.468489 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 22:44:32.470365 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.470346 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:44:32.489447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.477295 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:32.495482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.495457 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:44:32.511619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.498369 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 22:44:32.622930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.600734 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:32.726098 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.726007 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 22:44:32.738613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.738584 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 22:44:32.741530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.741505 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:44:32.741796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.741781 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:44:32.742033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.741986 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:44:32.778833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.777321 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" event={"ID":"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e","Type":"ContainerStarted","Data":"f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8"} Nov 03 22:44:32.778833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:32.777652 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:33.259508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:33.259468 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 22:44:33.259829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:33.259776 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 22:44:34.142669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:34.142628 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:34.143182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:34.142710 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:34.263530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:34.263495 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:34.265154 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:34.265132 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:34.286941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:34.286912 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:44:34.499998 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:34.499882 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" podStartSLOduration=10.499862133 podStartE2EDuration="10.499862133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:34.494015408 +0000 UTC m=+1255.116323722" watchObservedRunningTime="2026-11-03 22:44:34.499862133 +0000 UTC m=+1255.122170446" Nov 03 22:44:35.122861 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:35.117196 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:35.150824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:35.137189 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:35.150824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:35.137250 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:36.123605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:36.123570 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:36.124838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:36.124816 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:44:40.969788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:40.969754 5272 generic.go:356] "Generic (PLEG): container finished" podID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerID="1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f" exitCode=0 Nov 03 22:44:40.970250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:40.969832 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" event={"ID":"c33bd25b-b162-47bd-aae3-e523f3ee31a7","Type":"ContainerDied","Data":"1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f"} Nov 03 22:44:40.970296 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:40.970255 5272 scope.go:122] "RemoveContainer" containerID="1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f" Nov 03 22:44:40.971794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:40.971764 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:40.971881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:40.971856 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:40.971918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:40.971888 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:44:41.136082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.136049 5272 scope.go:122] "RemoveContainer" containerID="4576a026b45d9153eaae7fdd32cc84300d00e02a514cc7213b742bcd4f80ea23" Nov 03 22:44:41.150516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.150493 5272 scope.go:122] "RemoveContainer" containerID="c7cfe9151234802835599bc97c75b67eb3d2b8c0f801c369f05dcbec0ed67e90" Nov 03 22:44:41.160463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.160440 5272 scope.go:122] "RemoveContainer" containerID="30b2bf4c3b253612c4d46b9146478b9ea2e5cf569785042060e57c6b91d284e4" Nov 03 22:44:41.170477 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.170457 5272 scope.go:122] "RemoveContainer" containerID="2268bcf89acd5e0d4ac3fd4b9f5bc984f641f06dd64487333b73bce975e82cba" Nov 03 22:44:41.181084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.181062 5272 scope.go:122] "RemoveContainer" containerID="8b2203976084c504c3b0ac16fb844b27797abeadd204ff4c38a850f7d1a5bf0b" Nov 03 22:44:41.192292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.192259 5272 scope.go:122] "RemoveContainer" containerID="5c7de86600070a44a57ef990dfb3953d0057cabd88d5de8ebe83945d0021fd4c" Nov 03 22:44:41.205580 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.205556 5272 scope.go:122] "RemoveContainer" containerID="c8dcaed2faeb2ceabb8d04452d3ab56a572bcc1263a377085510516346bd61a1" Nov 03 22:44:41.216747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.216727 5272 scope.go:122] "RemoveContainer" containerID="bbad2d60c25c730311d1e1f70e9d26693048634d2c224337c2668790f5127178" Nov 03 22:44:41.229186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.229160 5272 scope.go:122] "RemoveContainer" containerID="bc06d225ec088fccc8cd6ea2852a11fa8ca10a4058393e15feed2a0adda636fe" Nov 03 22:44:41.239177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.239156 5272 scope.go:122] "RemoveContainer" containerID="ba8dc500ea1222f97eaaeec1df08061dfa91448a91f7a466a6f3d77acec82d73" Nov 03 22:44:41.598280 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.598248 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:44:41.981072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:41.980984 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" event={"ID":"c33bd25b-b162-47bd-aae3-e523f3ee31a7","Type":"ContainerStarted","Data":"712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4"} Nov 03 22:44:42.197612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:42.197578 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:44:42.199071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:42.199044 5272 patch_prober.go:29] interesting pod/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:42.199194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:42.199104 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 22:44:42.468867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:42.464917 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:44:43.198227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:43.198183 5272 patch_prober.go:29] interesting pod/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:43.198648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:43.198244 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 22:44:43.835675 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:43.835640 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:44:44.197629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:44.197533 5272 patch_prober.go:29] interesting pod/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:44.197629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:44.197594 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 22:44:45.197963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:45.197924 5272 patch_prober.go:29] interesting pod/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:45.198371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:45.197993 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 22:44:46.198339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:46.198300 5272 patch_prober.go:29] interesting pod/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:46.198742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:46.198369 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 22:44:47.202301 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:47.202271 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:44:47.203581 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:44:47.203554 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:44:54.503482 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:44:54.503395 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:45:00.169927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.169891 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn"] Nov 03 22:45:00.175367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.175334 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb"] Nov 03 22:45:00.175531 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.175340 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.186862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.186831 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z"] Nov 03 22:45:00.204833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.198405 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.222415 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.222383 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.242826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.230632 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:45:00.244570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.244528 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9fbab71-53ea-449c-b01a-f1c6550a26c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.244821 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.244785 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5hb\" (UniqueName: \"kubernetes.io/projected/a7db9322-4a97-46de-b7f6-70ea93c51df1-kube-api-access-rz5hb\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.244997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.244978 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db9322-4a97-46de-b7f6-70ea93c51df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.245263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.245246 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8w9\" (UniqueName: \"kubernetes.io/projected/b9fbab71-53ea-449c-b01a-f1c6550a26c1-kube-api-access-7s8w9\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.266627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.266594 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:45:00.316541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.316111 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt"] Nov 03 22:45:00.321749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.321723 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.325079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.325041 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:45:00.337377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.337340 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp"] Nov 03 22:45:00.356396 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.346895 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8w9\" (UniqueName: \"kubernetes.io/projected/b9fbab71-53ea-449c-b01a-f1c6550a26c1-kube-api-access-7s8w9\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.356760 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.356732 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78msx\" (UniqueName: \"kubernetes.io/projected/425f548c-b952-4b9f-8864-0f05377b538d-kube-api-access-78msx\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.357074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.357048 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9fbab71-53ea-449c-b01a-f1c6550a26c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.357255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.357236 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425f548c-b952-4b9f-8864-0f05377b538d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.357395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.357376 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5hb\" (UniqueName: \"kubernetes.io/projected/a7db9322-4a97-46de-b7f6-70ea93c51df1-kube-api-access-rz5hb\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.357592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.357566 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db9322-4a97-46de-b7f6-70ea93c51df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.359614 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.359587 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db9322-4a97-46de-b7f6-70ea93c51df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.360432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.360409 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9fbab71-53ea-449c-b01a-f1c6550a26c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.529475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.475223 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8w9\" (UniqueName: \"kubernetes.io/projected/b9fbab71-53ea-449c-b01a-f1c6550a26c1-kube-api-access-7s8w9\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.529475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.485391 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78msx\" (UniqueName: \"kubernetes.io/projected/425f548c-b952-4b9f-8864-0f05377b538d-kube-api-access-78msx\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.529475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.485653 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425f548c-b952-4b9f-8864-0f05377b538d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.529475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.520954 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5hb\" (UniqueName: \"kubernetes.io/projected/a7db9322-4a97-46de-b7f6-70ea93c51df1-kube-api-access-rz5hb\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.574499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.563919 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425f548c-b952-4b9f-8864-0f05377b538d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.574499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.567004 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:00.609881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.596262 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:00.633329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.626888 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k"] Nov 03 22:45:00.636239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.635233 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:00.704269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.704226 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scskf\" (UniqueName: \"kubernetes.io/projected/91e1c200-25f1-443d-9cb4-c4bcf283038e-kube-api-access-scskf\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:00.704479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.704349 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41dd3f6-f48d-428e-af46-c8ba2ce71517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.704558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.704507 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1c200-25f1-443d-9cb4-c4bcf283038e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:00.704678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.704657 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4msr\" (UniqueName: \"kubernetes.io/projected/f41dd3f6-f48d-428e-af46-c8ba2ce71517-kube-api-access-f4msr\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.752214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.752171 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5"] Nov 03 22:45:00.753890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.753780 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:00.813669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.788402 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78msx\" (UniqueName: \"kubernetes.io/projected/425f548c-b952-4b9f-8864-0f05377b538d-kube-api-access-78msx\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.813669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.788879 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx"] Nov 03 22:45:00.813669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.791023 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818356 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4msr\" (UniqueName: \"kubernetes.io/projected/f41dd3f6-f48d-428e-af46-c8ba2ce71517-kube-api-access-f4msr\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818474 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdc7\" (UniqueName: \"kubernetes.io/projected/80f86136-0a0b-44ba-806c-df4a42538a3f-kube-api-access-sfdc7\") pod \"maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818525 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818569 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scskf\" (UniqueName: \"kubernetes.io/projected/91e1c200-25f1-443d-9cb4-c4bcf283038e-kube-api-access-scskf\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818616 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ss68\" (UniqueName: \"kubernetes.io/projected/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-kube-api-access-2ss68\") pod \"maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818678 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41dd3f6-f48d-428e-af46-c8ba2ce71517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818718 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f86136-0a0b-44ba-806c-df4a42538a3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.818860 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1c200-25f1-443d-9cb4-c4bcf283038e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:00.845827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.822370 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41dd3f6-f48d-428e-af46-c8ba2ce71517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.867916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.853209 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1c200-25f1-443d-9cb4-c4bcf283038e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:00.867916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.854541 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn"] Nov 03 22:45:00.867916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.854589 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z"] Nov 03 22:45:00.867916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.854723 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:00.919841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.919779 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:00.942116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.935134 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4msr\" (UniqueName: \"kubernetes.io/projected/f41dd3f6-f48d-428e-af46-c8ba2ce71517-kube-api-access-f4msr\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.960192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.959743 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb"] Nov 03 22:45:00.960192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.960119 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:00.966609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.966541 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdc7\" (UniqueName: \"kubernetes.io/projected/80f86136-0a0b-44ba-806c-df4a42538a3f-kube-api-access-sfdc7\") pod \"maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:00.969027 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.969002 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:00.969679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.969645 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ss68\" (UniqueName: \"kubernetes.io/projected/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-kube-api-access-2ss68\") pod \"maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:00.969930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.969910 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f86136-0a0b-44ba-806c-df4a42538a3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:00.971906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.971875 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:00.973497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.973437 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f86136-0a0b-44ba-806c-df4a42538a3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:00.998976 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:00.998903 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scskf\" (UniqueName: \"kubernetes.io/projected/91e1c200-25f1-443d-9cb4-c4bcf283038e-kube-api-access-scskf\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:01.011631 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.011592 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt"] Nov 03 22:45:01.026136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.026074 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp"] Nov 03 22:45:01.035096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.034365 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx"] Nov 03 22:45:01.043570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.043524 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdc7\" (UniqueName: \"kubernetes.io/projected/80f86136-0a0b-44ba-806c-df4a42538a3f-kube-api-access-sfdc7\") pod \"maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:01.054234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.054209 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:01.066920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.059370 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:45:01.072964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.072931 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09630af1-91d7-49d3-92b6-e3516312596c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-pxtmx\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:01.073255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.073229 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589cj\" (UniqueName: \"kubernetes.io/projected/09630af1-91d7-49d3-92b6-e3516312596c-kube-api-access-589cj\") pod \"maas-api-key-cleanup-29895765-pxtmx\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:01.074585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.074563 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ss68\" (UniqueName: \"kubernetes.io/projected/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-kube-api-access-2ss68\") pod \"maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:01.084071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.084046 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k"] Nov 03 22:45:01.119010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.118611 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5"] Nov 03 22:45:01.168084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.167756 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:01.191311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.191275 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09630af1-91d7-49d3-92b6-e3516312596c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-pxtmx\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:01.191939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.191907 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589cj\" (UniqueName: \"kubernetes.io/projected/09630af1-91d7-49d3-92b6-e3516312596c-kube-api-access-589cj\") pod \"maas-api-key-cleanup-29895765-pxtmx\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:01.194425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.194392 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09630af1-91d7-49d3-92b6-e3516312596c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895765-pxtmx\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:01.210544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.196060 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:01.398763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.398720 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589cj\" (UniqueName: \"kubernetes.io/projected/09630af1-91d7-49d3-92b6-e3516312596c-kube-api-access-589cj\") pod \"maas-api-key-cleanup-29895765-pxtmx\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:01.483471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.483439 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:01.703185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.703120 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn"] Nov 03 22:45:01.723916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:01.723882 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb"] Nov 03 22:45:02.027919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:02.027833 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt"] Nov 03 22:45:02.257885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:02.254995 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5"] Nov 03 22:45:02.412441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:02.412396 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k"] Nov 03 22:45:02.629193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:02.629018 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z"] Nov 03 22:45:02.637284 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:02.637246 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx"] Nov 03 22:45:02.648533 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:45:02.648503 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffae27a_9b94_4cce_a0b4_b6946aa0612d.slice/crio-b55a8c535fa37eabed278b95a9554e8ad05dc7750af4a4193cd282e5250563a0 WatchSource:0}: Error finding container b55a8c535fa37eabed278b95a9554e8ad05dc7750af4a4193cd282e5250563a0: Status 404 returned error can't find the container with id b55a8c535fa37eabed278b95a9554e8ad05dc7750af4a4193cd282e5250563a0 Nov 03 22:45:02.655575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:02.655547 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp"] Nov 03 22:45:02.687420 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:45:02.687377 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e1c200_25f1_443d_9cb4_c4bcf283038e.slice/crio-33d77ce54fefa3159d521fc0a4a2f8ce5f57405c2a5d98aa278d953b7788185a WatchSource:0}: Error finding container 33d77ce54fefa3159d521fc0a4a2f8ce5f57405c2a5d98aa278d953b7788185a: Status 404 returned error can't find the container with id 33d77ce54fefa3159d521fc0a4a2f8ce5f57405c2a5d98aa278d953b7788185a Nov 03 22:45:03.060843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.045955 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" event={"ID":"a7db9322-4a97-46de-b7f6-70ea93c51df1","Type":"ContainerStarted","Data":"cbc6777017b5cda8f3a6ce5ccfe00435f225267fecc0f485e1306490cd1aa6e8"} Nov 03 22:45:03.091210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.091167 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" event={"ID":"b9fbab71-53ea-449c-b01a-f1c6550a26c1","Type":"ContainerStarted","Data":"513ab33d4ad19f849cfe155bab325c7e17730e9d2897bb0cc33df9dcdcc48572"} Nov 03 22:45:03.136651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.136494 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerStarted","Data":"51383edde7237d8f5ea3b0bf1afc4e0566255385fb33c307f75a91636f4209e3"} Nov 03 22:45:03.138826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.138732 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" event={"ID":"09630af1-91d7-49d3-92b6-e3516312596c","Type":"ContainerStarted","Data":"c5f90758b5f4b49e8e441d391f5dcacabf803925d0c8332f585bf6dd043f0125"} Nov 03 22:45:03.292042 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.289182 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.446653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.422606 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.449383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.447846 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.569056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.502412 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 22:45:03.569056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.522924 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" podStartSLOduration=3.522901342 podStartE2EDuration="3.522901342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:03.297904422 +0000 UTC m=+1283.920212731" watchObservedRunningTime="2026-11-03 22:45:03.522901342 +0000 UTC m=+1284.145209651" Nov 03 22:45:03.569056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.526220 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.569056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.537961 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.777866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.770360 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:03.844012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:03.799892 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.204043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.203721 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.331229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.320964 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" event={"ID":"1ffae27a-9b94-4cce-a0b4-b6946aa0612d","Type":"ContainerStarted","Data":"b55a8c535fa37eabed278b95a9554e8ad05dc7750af4a4193cd282e5250563a0"} Nov 03 22:45:04.358757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.355297 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.430565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.428642 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 22:45:04.442857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.432781 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:45:04.442857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.441237 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" event={"ID":"a7db9322-4a97-46de-b7f6-70ea93c51df1","Type":"ContainerDied","Data":"8bc0ebff142d67bc27740549149f9a1036056e02c5a807ccd8a0e53d1615de3f"} Nov 03 22:45:04.442857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.433562 5272 generic.go:356] "Generic (PLEG): container finished" podID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerID="8bc0ebff142d67bc27740549149f9a1036056e02c5a807ccd8a0e53d1615de3f" exitCode=60 Nov 03 22:45:04.444507 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.434217 5272 scope.go:122] "RemoveContainer" containerID="8bc0ebff142d67bc27740549149f9a1036056e02c5a807ccd8a0e53d1615de3f" Nov 03 22:45:04.548352 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.513164 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.552827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.526889 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.553655 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.553620 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.594119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.565873 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:45:04.594119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.585311 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:45:04.594119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.585379 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:45:04.618056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.616192 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" event={"ID":"91e1c200-25f1-443d-9cb4-c4bcf283038e","Type":"ContainerStarted","Data":"33d77ce54fefa3159d521fc0a4a2f8ce5f57405c2a5d98aa278d953b7788185a"} Nov 03 22:45:04.923003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.779761 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:04.923003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.922976 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:45:04.935794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.927223 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 03 22:45:05.135941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.967167 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.135941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.992659 5272 generic.go:356] "Generic (PLEG): container finished" podID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerID="d1d96f9daf69f585144479e5c032d349c6ff586182d1ef5c56524609219386cd" exitCode=60 Nov 03 22:45:05.160828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:04.992689 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" event={"ID":"b9fbab71-53ea-449c-b01a-f1c6550a26c1","Type":"ContainerDied","Data":"d1d96f9daf69f585144479e5c032d349c6ff586182d1ef5c56524609219386cd"} Nov 03 22:45:05.160828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.016251 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.160828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.026455 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.160828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.126206 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.265073 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.264962 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.362917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.361796 5272 scope.go:122] "RemoveContainer" containerID="d1d96f9daf69f585144479e5c032d349c6ff586182d1ef5c56524609219386cd" Nov 03 22:45:05.398101 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.391777 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:05.398101 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.391858 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.444822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.429466 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" event={"ID":"425f548c-b952-4b9f-8864-0f05377b538d","Type":"ContainerStarted","Data":"ca06418ffe44573e8dc2136f9270895010ca95e50d1632befe9a748f533185dc"} Nov 03 22:45:05.455845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.455028 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 22:45:05.501938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.492650 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerStarted","Data":"6d2a9bc1a97cc460d4d7280f4a5ee6be41a97c08b76a420071c13047b703257a"} Nov 03 22:45:05.503145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.503100 5272 scope.go:122] "RemoveContainer" containerID="6d2a9bc1a97cc460d4d7280f4a5ee6be41a97c08b76a420071c13047b703257a" Nov 03 22:45:05.543993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.543951 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 22:45:05.582552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.582513 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.618875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.608994 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.619225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.619193 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 22:45:05.806881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.804669 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" event={"ID":"80f86136-0a0b-44ba-806c-df4a42538a3f","Type":"ContainerStarted","Data":"c99db4c65fc8b459d51df1ff99d0c4b830283635b7c64bbb13f4a8922635667a"} Nov 03 22:45:05.810605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.810576 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 22:45:05.905883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.905792 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 22:45:05.906857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:05.906403 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 22:45:06.315578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.315506 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" podStartSLOduration=6.315487446 podStartE2EDuration="6.315487446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.312634344 +0000 UTC m=+1286.934942652" watchObservedRunningTime="2026-11-03 22:45:06.315487446 +0000 UTC m=+1286.937795756" Nov 03 22:45:06.755715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.755680 5272 generic.go:356] "Generic (PLEG): container finished" podID="09630af1-91d7-49d3-92b6-e3516312596c" containerID="2d7647c7e5a5b5cc767d2af18a39cbf6af2b84ee5bcfda784ade67108834eb59" exitCode=60 Nov 03 22:45:06.802958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.802873 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" event={"ID":"09630af1-91d7-49d3-92b6-e3516312596c","Type":"ContainerDied","Data":"2d7647c7e5a5b5cc767d2af18a39cbf6af2b84ee5bcfda784ade67108834eb59"} Nov 03 22:45:06.805313 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.805289 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerID="857f8bbfda19130bdf2ec5c1b865c7d3a982eb833c5f7bf3a71f444035b42951" exitCode=60 Nov 03 22:45:06.805570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.805553 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" event={"ID":"1ffae27a-9b94-4cce-a0b4-b6946aa0612d","Type":"ContainerDied","Data":"857f8bbfda19130bdf2ec5c1b865c7d3a982eb833c5f7bf3a71f444035b42951"} Nov 03 22:45:06.806071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.806052 5272 scope.go:122] "RemoveContainer" containerID="857f8bbfda19130bdf2ec5c1b865c7d3a982eb833c5f7bf3a71f444035b42951" Nov 03 22:45:06.808036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.807975 5272 scope.go:122] "RemoveContainer" containerID="2d7647c7e5a5b5cc767d2af18a39cbf6af2b84ee5bcfda784ade67108834eb59" Nov 03 22:45:06.894613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.893898 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" event={"ID":"a7db9322-4a97-46de-b7f6-70ea93c51df1","Type":"ContainerStarted","Data":"e65306acda0fa3a2c3f8496006cbaabf2cd775a1252e67506507d39860341e77"} Nov 03 22:45:06.897525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.897495 5272 generic.go:356] "Generic (PLEG): container finished" podID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerID="8d7ccc555fb657818b559aa84fe572b6e70a6993d553175f51b668809d2a34fd" exitCode=60 Nov 03 22:45:06.898273 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.898056 5272 scope.go:122] "RemoveContainer" containerID="8d7ccc555fb657818b559aa84fe572b6e70a6993d553175f51b668809d2a34fd" Nov 03 22:45:06.902687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.898081 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" event={"ID":"91e1c200-25f1-443d-9cb4-c4bcf283038e","Type":"ContainerDied","Data":"8d7ccc555fb657818b559aa84fe572b6e70a6993d553175f51b668809d2a34fd"} Nov 03 22:45:06.903255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.903218 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" event={"ID":"b9fbab71-53ea-449c-b01a-f1c6550a26c1","Type":"ContainerStarted","Data":"0dc3c173bf033764e8a952634b7e044f8e372fc6060e6931b6662917ef23ebd0"} Nov 03 22:45:06.905233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.905210 5272 generic.go:356] "Generic (PLEG): container finished" podID="425f548c-b952-4b9f-8864-0f05377b538d" containerID="c19a894ac1de04caa8a8bfc5d01e6608133351a82412a67ef44e4d843fbe8abd" exitCode=60 Nov 03 22:45:06.907329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.905733 5272 scope.go:122] "RemoveContainer" containerID="c19a894ac1de04caa8a8bfc5d01e6608133351a82412a67ef44e4d843fbe8abd" Nov 03 22:45:06.908253 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.906455 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" event={"ID":"425f548c-b952-4b9f-8864-0f05377b538d","Type":"ContainerDied","Data":"c19a894ac1de04caa8a8bfc5d01e6608133351a82412a67ef44e4d843fbe8abd"} Nov 03 22:45:06.911758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.910904 5272 generic.go:356] "Generic (PLEG): container finished" podID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerID="6d2a9bc1a97cc460d4d7280f4a5ee6be41a97c08b76a420071c13047b703257a" exitCode=60 Nov 03 22:45:06.911758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.910983 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerDied","Data":"6d2a9bc1a97cc460d4d7280f4a5ee6be41a97c08b76a420071c13047b703257a"} Nov 03 22:45:06.911758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.911020 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerStarted","Data":"c28ec007f7daf89ab2ec423fbc8dcfc776a8d33aff9b307f5e55ae537568a8f5"} Nov 03 22:45:06.914632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.914610 5272 generic.go:356] "Generic (PLEG): container finished" podID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerID="6733f5cbd507b15b5bb1385cb7f827b6c7355519e79729cc23d1498da8d504ff" exitCode=60 Nov 03 22:45:06.915577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.915562 5272 scope.go:122] "RemoveContainer" containerID="6733f5cbd507b15b5bb1385cb7f827b6c7355519e79729cc23d1498da8d504ff" Nov 03 22:45:06.917361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:06.917339 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" event={"ID":"80f86136-0a0b-44ba-806c-df4a42538a3f","Type":"ContainerDied","Data":"6733f5cbd507b15b5bb1385cb7f827b6c7355519e79729cc23d1498da8d504ff"} Nov 03 22:45:07.299894 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.299842 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" podStartSLOduration=7.299828417 podStartE2EDuration="7.299828417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:07.29632411 +0000 UTC m=+1287.918632417" watchObservedRunningTime="2026-11-03 22:45:07.299828417 +0000 UTC m=+1287.922136722" Nov 03 22:45:07.521030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.520978 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" podStartSLOduration=7.52096383 podStartE2EDuration="7.52096383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:07.513879762 +0000 UTC m=+1288.136188073" watchObservedRunningTime="2026-11-03 22:45:07.52096383 +0000 UTC m=+1288.143272168" Nov 03 22:45:07.937998 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.937962 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" event={"ID":"09630af1-91d7-49d3-92b6-e3516312596c","Type":"ContainerStarted","Data":"8c9e0c5ef6b34bf1513995b31e1e80c5cb6385284211fdfa4ca1010daf18d4d2"} Nov 03 22:45:07.940187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.940165 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" event={"ID":"1ffae27a-9b94-4cce-a0b4-b6946aa0612d","Type":"ContainerStarted","Data":"1e6dc1e0edf1dedb9362146a7dddcead482d7d89da7ad58dc051160e66786129"} Nov 03 22:45:07.942245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.942222 5272 generic.go:356] "Generic (PLEG): container finished" podID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerID="e65306acda0fa3a2c3f8496006cbaabf2cd775a1252e67506507d39860341e77" exitCode=60 Nov 03 22:45:07.942499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.942298 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" event={"ID":"a7db9322-4a97-46de-b7f6-70ea93c51df1","Type":"ContainerDied","Data":"e65306acda0fa3a2c3f8496006cbaabf2cd775a1252e67506507d39860341e77"} Nov 03 22:45:07.942499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.942348 5272 scope.go:122] "RemoveContainer" containerID="8bc0ebff142d67bc27740549149f9a1036056e02c5a807ccd8a0e53d1615de3f" Nov 03 22:45:07.942610 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.942502 5272 scope.go:122] "RemoveContainer" containerID="e65306acda0fa3a2c3f8496006cbaabf2cd775a1252e67506507d39860341e77" Nov 03 22:45:07.942859 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:07.942834 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn_odh-ai-gateway-infra(a7db9322-4a97-46de-b7f6-70ea93c51df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" Nov 03 22:45:07.949720 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.949699 5272 generic.go:356] "Generic (PLEG): container finished" podID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerID="0dc3c173bf033764e8a952634b7e044f8e372fc6060e6931b6662917ef23ebd0" exitCode=60 Nov 03 22:45:07.949897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.949786 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" event={"ID":"b9fbab71-53ea-449c-b01a-f1c6550a26c1","Type":"ContainerDied","Data":"0dc3c173bf033764e8a952634b7e044f8e372fc6060e6931b6662917ef23ebd0"} Nov 03 22:45:07.950059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.950041 5272 scope.go:122] "RemoveContainer" containerID="0dc3c173bf033764e8a952634b7e044f8e372fc6060e6931b6662917ef23ebd0" Nov 03 22:45:07.950309 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:07.950288 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb_odh-ai-gateway-infra(b9fbab71-53ea-449c-b01a-f1c6550a26c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" Nov 03 22:45:07.952079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.952056 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" event={"ID":"425f548c-b952-4b9f-8864-0f05377b538d","Type":"ContainerStarted","Data":"70a7089c7b5c8624139756c27f39a5dd5378215bf13595ebf41c5fafe69634ff"} Nov 03 22:45:07.954134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.954111 5272 generic.go:356] "Generic (PLEG): container finished" podID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerID="c28ec007f7daf89ab2ec423fbc8dcfc776a8d33aff9b307f5e55ae537568a8f5" exitCode=60 Nov 03 22:45:07.954214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.954190 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerDied","Data":"c28ec007f7daf89ab2ec423fbc8dcfc776a8d33aff9b307f5e55ae537568a8f5"} Nov 03 22:45:07.954411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.954395 5272 scope.go:122] "RemoveContainer" containerID="c28ec007f7daf89ab2ec423fbc8dcfc776a8d33aff9b307f5e55ae537568a8f5" Nov 03 22:45:07.954680 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:07.954658 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt_odh-ai-gateway-infra(f41dd3f6-f48d-428e-af46-c8ba2ce71517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" Nov 03 22:45:07.956015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.955996 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" event={"ID":"80f86136-0a0b-44ba-806c-df4a42538a3f","Type":"ContainerStarted","Data":"fdad60d6c64f017038ab9c6f5b6b70eda3fef236dc885b55e91c1d283fd1bf83"} Nov 03 22:45:07.958794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:07.958774 5272 scope.go:122] "RemoveContainer" containerID="d1d96f9daf69f585144479e5c032d349c6ff586182d1ef5c56524609219386cd" Nov 03 22:45:08.001590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.001488 5272 scope.go:122] "RemoveContainer" containerID="6d2a9bc1a97cc460d4d7280f4a5ee6be41a97c08b76a420071c13047b703257a" Nov 03 22:45:08.050234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.050182 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" podStartSLOduration=8.050168363 podStartE2EDuration="8.050168363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:08.046884178 +0000 UTC m=+1288.669192497" watchObservedRunningTime="2026-11-03 22:45:08.050168363 +0000 UTC m=+1288.672476671" Nov 03 22:45:08.373001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.372952 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" podStartSLOduration=8.37293725 podStartE2EDuration="8.37293725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:08.369750254 +0000 UTC m=+1288.992058563" watchObservedRunningTime="2026-11-03 22:45:08.37293725 +0000 UTC m=+1288.995245583" Nov 03 22:45:08.482203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.482148 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" podStartSLOduration=8.482132831 podStartE2EDuration="8.482132831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:08.477621065 +0000 UTC m=+1289.099929372" watchObservedRunningTime="2026-11-03 22:45:08.482132831 +0000 UTC m=+1289.104441142" Nov 03 22:45:08.692918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.692788 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" podStartSLOduration=8.692773645 podStartE2EDuration="8.692773645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:08.691747662 +0000 UTC m=+1289.314055971" watchObservedRunningTime="2026-11-03 22:45:08.692773645 +0000 UTC m=+1289.315081953" Nov 03 22:45:08.966502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.966408 5272 generic.go:356] "Generic (PLEG): container finished" podID="425f548c-b952-4b9f-8864-0f05377b538d" containerID="70a7089c7b5c8624139756c27f39a5dd5378215bf13595ebf41c5fafe69634ff" exitCode=60 Nov 03 22:45:08.966502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.966488 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" event={"ID":"425f548c-b952-4b9f-8864-0f05377b538d","Type":"ContainerDied","Data":"70a7089c7b5c8624139756c27f39a5dd5378215bf13595ebf41c5fafe69634ff"} Nov 03 22:45:08.967038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.966557 5272 scope.go:122] "RemoveContainer" containerID="c19a894ac1de04caa8a8bfc5d01e6608133351a82412a67ef44e4d843fbe8abd" Nov 03 22:45:08.967038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.966846 5272 scope.go:122] "RemoveContainer" containerID="70a7089c7b5c8624139756c27f39a5dd5378215bf13595ebf41c5fafe69634ff" Nov 03 22:45:08.967208 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.967184 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z_odh-ai-gateway-infra(425f548c-b952-4b9f-8864-0f05377b538d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" podUID="425f548c-b952-4b9f-8864-0f05377b538d" Nov 03 22:45:08.968696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.968670 5272 scope.go:122] "RemoveContainer" containerID="c28ec007f7daf89ab2ec423fbc8dcfc776a8d33aff9b307f5e55ae537568a8f5" Nov 03 22:45:08.968973 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.968952 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt_odh-ai-gateway-infra(f41dd3f6-f48d-428e-af46-c8ba2ce71517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" Nov 03 22:45:08.970328 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.970309 5272 generic.go:356] "Generic (PLEG): container finished" podID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerID="fdad60d6c64f017038ab9c6f5b6b70eda3fef236dc885b55e91c1d283fd1bf83" exitCode=60 Nov 03 22:45:08.970429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.970354 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" event={"ID":"80f86136-0a0b-44ba-806c-df4a42538a3f","Type":"ContainerDied","Data":"fdad60d6c64f017038ab9c6f5b6b70eda3fef236dc885b55e91c1d283fd1bf83"} Nov 03 22:45:08.970629 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.970613 5272 scope.go:122] "RemoveContainer" containerID="fdad60d6c64f017038ab9c6f5b6b70eda3fef236dc885b55e91c1d283fd1bf83" Nov 03 22:45:08.970930 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.970906 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k_odh-ai-gateway-infra(80f86136-0a0b-44ba-806c-df4a42538a3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" Nov 03 22:45:08.972416 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.972397 5272 generic.go:356] "Generic (PLEG): container finished" podID="09630af1-91d7-49d3-92b6-e3516312596c" containerID="8c9e0c5ef6b34bf1513995b31e1e80c5cb6385284211fdfa4ca1010daf18d4d2" exitCode=60 Nov 03 22:45:08.972529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.972467 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" event={"ID":"09630af1-91d7-49d3-92b6-e3516312596c","Type":"ContainerDied","Data":"8c9e0c5ef6b34bf1513995b31e1e80c5cb6385284211fdfa4ca1010daf18d4d2"} Nov 03 22:45:08.972710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.972695 5272 scope.go:122] "RemoveContainer" containerID="8c9e0c5ef6b34bf1513995b31e1e80c5cb6385284211fdfa4ca1010daf18d4d2" Nov 03 22:45:08.973014 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.972992 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-pxtmx_odh-ai-gateway-infra(09630af1-91d7-49d3-92b6-e3516312596c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" podUID="09630af1-91d7-49d3-92b6-e3516312596c" Nov 03 22:45:08.974609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.974588 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerID="1e6dc1e0edf1dedb9362146a7dddcead482d7d89da7ad58dc051160e66786129" exitCode=60 Nov 03 22:45:08.974693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.974658 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" event={"ID":"1ffae27a-9b94-4cce-a0b4-b6946aa0612d","Type":"ContainerDied","Data":"1e6dc1e0edf1dedb9362146a7dddcead482d7d89da7ad58dc051160e66786129"} Nov 03 22:45:08.974934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.974914 5272 scope.go:122] "RemoveContainer" containerID="1e6dc1e0edf1dedb9362146a7dddcead482d7d89da7ad58dc051160e66786129" Nov 03 22:45:08.975190 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.975169 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5_odh-ai-gateway-infra(1ffae27a-9b94-4cce-a0b4-b6946aa0612d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" Nov 03 22:45:08.976910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.976893 5272 scope.go:122] "RemoveContainer" containerID="e65306acda0fa3a2c3f8496006cbaabf2cd775a1252e67506507d39860341e77" Nov 03 22:45:08.977174 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.977156 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn_odh-ai-gateway-infra(a7db9322-4a97-46de-b7f6-70ea93c51df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" Nov 03 22:45:08.978549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.978532 5272 generic.go:356] "Generic (PLEG): container finished" podID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerID="9e2464c5bd9bb5366a3a5e5f86debaeab3f5e65ca27070c82abd6897ccb3ec67" exitCode=60 Nov 03 22:45:08.978648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.978608 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" event={"ID":"91e1c200-25f1-443d-9cb4-c4bcf283038e","Type":"ContainerDied","Data":"9e2464c5bd9bb5366a3a5e5f86debaeab3f5e65ca27070c82abd6897ccb3ec67"} Nov 03 22:45:08.978954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.978937 5272 scope.go:122] "RemoveContainer" containerID="9e2464c5bd9bb5366a3a5e5f86debaeab3f5e65ca27070c82abd6897ccb3ec67" Nov 03 22:45:08.979208 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.979178 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp_odh-ai-gateway-infra(91e1c200-25f1-443d-9cb4-c4bcf283038e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" Nov 03 22:45:08.980337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.980313 5272 scope.go:122] "RemoveContainer" containerID="0dc3c173bf033764e8a952634b7e044f8e372fc6060e6931b6662917ef23ebd0" Nov 03 22:45:08.980551 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:08.980531 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb_odh-ai-gateway-infra(b9fbab71-53ea-449c-b01a-f1c6550a26c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" Nov 03 22:45:08.982234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.982214 5272 scope.go:122] "RemoveContainer" containerID="6733f5cbd507b15b5bb1385cb7f827b6c7355519e79729cc23d1498da8d504ff" Nov 03 22:45:08.995927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:08.995904 5272 scope.go:122] "RemoveContainer" containerID="2d7647c7e5a5b5cc767d2af18a39cbf6af2b84ee5bcfda784ade67108834eb59" Nov 03 22:45:09.008875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:09.008858 5272 scope.go:122] "RemoveContainer" containerID="857f8bbfda19130bdf2ec5c1b865c7d3a982eb833c5f7bf3a71f444035b42951" Nov 03 22:45:09.025021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:09.025002 5272 scope.go:122] "RemoveContainer" containerID="8d7ccc555fb657818b559aa84fe572b6e70a6993d553175f51b668809d2a34fd" Nov 03 22:45:09.992627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:09.992594 5272 scope.go:122] "RemoveContainer" containerID="fdad60d6c64f017038ab9c6f5b6b70eda3fef236dc885b55e91c1d283fd1bf83" Nov 03 22:45:09.993111 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:09.993048 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k_odh-ai-gateway-infra(80f86136-0a0b-44ba-806c-df4a42538a3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" Nov 03 22:45:09.994428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:09.994400 5272 scope.go:122] "RemoveContainer" containerID="8c9e0c5ef6b34bf1513995b31e1e80c5cb6385284211fdfa4ca1010daf18d4d2" Nov 03 22:45:09.994769 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:09.994742 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-pxtmx_odh-ai-gateway-infra(09630af1-91d7-49d3-92b6-e3516312596c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" podUID="09630af1-91d7-49d3-92b6-e3516312596c" Nov 03 22:45:09.996121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:09.996100 5272 scope.go:122] "RemoveContainer" containerID="1e6dc1e0edf1dedb9362146a7dddcead482d7d89da7ad58dc051160e66786129" Nov 03 22:45:09.996449 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:09.996426 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5_odh-ai-gateway-infra(1ffae27a-9b94-4cce-a0b4-b6946aa0612d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" Nov 03 22:45:09.999380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:09.999360 5272 scope.go:122] "RemoveContainer" containerID="70a7089c7b5c8624139756c27f39a5dd5378215bf13595ebf41c5fafe69634ff" Nov 03 22:45:09.999661 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:09.999642 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z_odh-ai-gateway-infra(425f548c-b952-4b9f-8864-0f05377b538d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" podUID="425f548c-b952-4b9f-8864-0f05377b538d" Nov 03 22:45:19.493569 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:19.493542 5272 scope.go:122] "RemoveContainer" containerID="e65306acda0fa3a2c3f8496006cbaabf2cd775a1252e67506507d39860341e77" Nov 03 22:45:20.194113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:20.194077 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" event={"ID":"a7db9322-4a97-46de-b7f6-70ea93c51df1","Type":"ContainerStarted","Data":"c79328c0d3ccfc55788487ace7235d8f00a7d86567937b0b88e6bbbab305792a"} Nov 03 22:45:20.485926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:20.485899 5272 scope.go:122] "RemoveContainer" containerID="fdad60d6c64f017038ab9c6f5b6b70eda3fef236dc885b55e91c1d283fd1bf83" Nov 03 22:45:20.486055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:20.486010 5272 scope.go:122] "RemoveContainer" containerID="c28ec007f7daf89ab2ec423fbc8dcfc776a8d33aff9b307f5e55ae537568a8f5" Nov 03 22:45:20.486334 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:20.486314 5272 scope.go:122] "RemoveContainer" containerID="70a7089c7b5c8624139756c27f39a5dd5378215bf13595ebf41c5fafe69634ff" Nov 03 22:45:20.486654 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:20.486636 5272 scope.go:122] "RemoveContainer" containerID="0dc3c173bf033764e8a952634b7e044f8e372fc6060e6931b6662917ef23ebd0" Nov 03 22:45:21.205364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.205321 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" event={"ID":"b9fbab71-53ea-449c-b01a-f1c6550a26c1","Type":"ContainerStarted","Data":"079ed9e7d0ae5e645b8fe3b3c94dfbadc7d75c406010afe70f72dca3ffac5844"} Nov 03 22:45:21.207602 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.207572 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" event={"ID":"425f548c-b952-4b9f-8864-0f05377b538d","Type":"ContainerStarted","Data":"a9b2d5479929ffd4cc27652b2d08486e05f2d35fdfcfee5263c51c200b431a38"} Nov 03 22:45:21.209697 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.209674 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerStarted","Data":"98ed51bddba3c85c7cde6299d2f02bb3777d96ce41c9f7c63e5d504f1a1159df"} Nov 03 22:45:21.211844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.211786 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" event={"ID":"80f86136-0a0b-44ba-806c-df4a42538a3f","Type":"ContainerStarted","Data":"677509f642ab398da610ac8eeea5868d98d7fcee87adfbd47b6a989b779c9a51"} Nov 03 22:45:21.213882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.213862 5272 generic.go:356] "Generic (PLEG): container finished" podID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerID="c79328c0d3ccfc55788487ace7235d8f00a7d86567937b0b88e6bbbab305792a" exitCode=60 Nov 03 22:45:21.213974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.213897 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" event={"ID":"a7db9322-4a97-46de-b7f6-70ea93c51df1","Type":"ContainerDied","Data":"c79328c0d3ccfc55788487ace7235d8f00a7d86567937b0b88e6bbbab305792a"} Nov 03 22:45:21.213974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.213956 5272 scope.go:122] "RemoveContainer" containerID="e65306acda0fa3a2c3f8496006cbaabf2cd775a1252e67506507d39860341e77" Nov 03 22:45:21.214262 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.214239 5272 scope.go:122] "RemoveContainer" containerID="c79328c0d3ccfc55788487ace7235d8f00a7d86567937b0b88e6bbbab305792a" Nov 03 22:45:21.214460 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:21.214439 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn_odh-ai-gateway-infra(a7db9322-4a97-46de-b7f6-70ea93c51df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" Nov 03 22:45:21.360908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.360827 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn"] Nov 03 22:45:21.496342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.496307 5272 scope.go:122] "RemoveContainer" containerID="1e6dc1e0edf1dedb9362146a7dddcead482d7d89da7ad58dc051160e66786129" Nov 03 22:45:21.496515 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.496484 5272 scope.go:122] "RemoveContainer" containerID="9e2464c5bd9bb5366a3a5e5f86debaeab3f5e65ca27070c82abd6897ccb3ec67" Nov 03 22:45:21.496657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:21.496638 5272 scope.go:122] "RemoveContainer" containerID="8c9e0c5ef6b34bf1513995b31e1e80c5cb6385284211fdfa4ca1010daf18d4d2" Nov 03 22:45:22.230847 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.230781 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" event={"ID":"91e1c200-25f1-443d-9cb4-c4bcf283038e","Type":"ContainerStarted","Data":"308b8fad93c68fa06afe221fcd24c6c6d2437c727463549acc49719e532edb9d"} Nov 03 22:45:22.233043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.233020 5272 generic.go:356] "Generic (PLEG): container finished" podID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerID="079ed9e7d0ae5e645b8fe3b3c94dfbadc7d75c406010afe70f72dca3ffac5844" exitCode=60 Nov 03 22:45:22.233169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.233094 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" event={"ID":"b9fbab71-53ea-449c-b01a-f1c6550a26c1","Type":"ContainerDied","Data":"079ed9e7d0ae5e645b8fe3b3c94dfbadc7d75c406010afe70f72dca3ffac5844"} Nov 03 22:45:22.233169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.233160 5272 scope.go:122] "RemoveContainer" containerID="0dc3c173bf033764e8a952634b7e044f8e372fc6060e6931b6662917ef23ebd0" Nov 03 22:45:22.233531 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.233516 5272 scope.go:122] "RemoveContainer" containerID="079ed9e7d0ae5e645b8fe3b3c94dfbadc7d75c406010afe70f72dca3ffac5844" Nov 03 22:45:22.233832 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:22.233789 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb_odh-ai-gateway-infra(b9fbab71-53ea-449c-b01a-f1c6550a26c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" Nov 03 22:45:22.235275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.235255 5272 generic.go:356] "Generic (PLEG): container finished" podID="425f548c-b952-4b9f-8864-0f05377b538d" containerID="a9b2d5479929ffd4cc27652b2d08486e05f2d35fdfcfee5263c51c200b431a38" exitCode=60 Nov 03 22:45:22.235361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.235319 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" event={"ID":"425f548c-b952-4b9f-8864-0f05377b538d","Type":"ContainerDied","Data":"a9b2d5479929ffd4cc27652b2d08486e05f2d35fdfcfee5263c51c200b431a38"} Nov 03 22:45:22.235634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.235620 5272 scope.go:122] "RemoveContainer" containerID="a9b2d5479929ffd4cc27652b2d08486e05f2d35fdfcfee5263c51c200b431a38" Nov 03 22:45:22.235934 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:22.235910 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z_odh-ai-gateway-infra(425f548c-b952-4b9f-8864-0f05377b538d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" podUID="425f548c-b952-4b9f-8864-0f05377b538d" Nov 03 22:45:22.237368 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.237349 5272 generic.go:356] "Generic (PLEG): container finished" podID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerID="98ed51bddba3c85c7cde6299d2f02bb3777d96ce41c9f7c63e5d504f1a1159df" exitCode=60 Nov 03 22:45:22.237474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.237424 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerDied","Data":"98ed51bddba3c85c7cde6299d2f02bb3777d96ce41c9f7c63e5d504f1a1159df"} Nov 03 22:45:22.237792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.237774 5272 scope.go:122] "RemoveContainer" containerID="98ed51bddba3c85c7cde6299d2f02bb3777d96ce41c9f7c63e5d504f1a1159df" Nov 03 22:45:22.238067 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:22.238045 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt_odh-ai-gateway-infra(f41dd3f6-f48d-428e-af46-c8ba2ce71517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" Nov 03 22:45:22.239721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.239703 5272 generic.go:356] "Generic (PLEG): container finished" podID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerID="677509f642ab398da610ac8eeea5868d98d7fcee87adfbd47b6a989b779c9a51" exitCode=60 Nov 03 22:45:22.239841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.239778 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" event={"ID":"80f86136-0a0b-44ba-806c-df4a42538a3f","Type":"ContainerDied","Data":"677509f642ab398da610ac8eeea5868d98d7fcee87adfbd47b6a989b779c9a51"} Nov 03 22:45:22.240103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.240084 5272 scope.go:122] "RemoveContainer" containerID="677509f642ab398da610ac8eeea5868d98d7fcee87adfbd47b6a989b779c9a51" Nov 03 22:45:22.240346 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:22.240323 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k_odh-ai-gateway-infra(80f86136-0a0b-44ba-806c-df4a42538a3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" Nov 03 22:45:22.242093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.242074 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" event={"ID":"09630af1-91d7-49d3-92b6-e3516312596c","Type":"ContainerStarted","Data":"50f093cb23d42cd01d77b0712ae567e4df8135272fe0334a95c4c1031e0e3723"} Nov 03 22:45:22.255424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.255400 5272 scope.go:122] "RemoveContainer" containerID="70a7089c7b5c8624139756c27f39a5dd5378215bf13595ebf41c5fafe69634ff" Nov 03 22:45:22.272525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.272506 5272 scope.go:122] "RemoveContainer" containerID="c28ec007f7daf89ab2ec423fbc8dcfc776a8d33aff9b307f5e55ae537568a8f5" Nov 03 22:45:22.288797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.288775 5272 scope.go:122] "RemoveContainer" containerID="fdad60d6c64f017038ab9c6f5b6b70eda3fef236dc885b55e91c1d283fd1bf83" Nov 03 22:45:22.330950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.330918 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb"] Nov 03 22:45:22.392071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.392038 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:22.437666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.437634 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k"] Nov 03 22:45:22.532349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.532319 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7db9322-4a97-46de-b7f6-70ea93c51df1-kube-api-access-rz5hb\" (UniqueName: \"kubernetes.io/projected/a7db9322-4a97-46de-b7f6-70ea93c51df1-kube-api-access-rz5hb\") pod \"a7db9322-4a97-46de-b7f6-70ea93c51df1\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " Nov 03 22:45:22.532479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.532376 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7db9322-4a97-46de-b7f6-70ea93c51df1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db9322-4a97-46de-b7f6-70ea93c51df1-maas-api-service-ca\") pod \"a7db9322-4a97-46de-b7f6-70ea93c51df1\" (UID: \"a7db9322-4a97-46de-b7f6-70ea93c51df1\") " Nov 03 22:45:22.532765 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.532736 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7db9322-4a97-46de-b7f6-70ea93c51df1-maas-api-service-ca" pod "a7db9322-4a97-46de-b7f6-70ea93c51df1" (UID: "a7db9322-4a97-46de-b7f6-70ea93c51df1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.533085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.533068 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db9322-4a97-46de-b7f6-70ea93c51df1-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.534489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.534467 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7db9322-4a97-46de-b7f6-70ea93c51df1-kube-api-access-rz5hb" pod "a7db9322-4a97-46de-b7f6-70ea93c51df1" (UID: "a7db9322-4a97-46de-b7f6-70ea93c51df1"). InnerVolumeSpecName "kube-api-access-rz5hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:22.545950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.545927 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt"] Nov 03 22:45:22.634854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.634817 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5hb\" (UniqueName: \"kubernetes.io/projected/a7db9322-4a97-46de-b7f6-70ea93c51df1-kube-api-access-rz5hb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.762421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:22.762390 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z"] Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.387442 5272 generic.go:356] "Generic (PLEG): container finished" podID="09630af1-91d7-49d3-92b6-e3516312596c" containerID="50f093cb23d42cd01d77b0712ae567e4df8135272fe0334a95c4c1031e0e3723" exitCode=60 Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.387760 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" event={"ID":"09630af1-91d7-49d3-92b6-e3516312596c","Type":"ContainerDied","Data":"50f093cb23d42cd01d77b0712ae567e4df8135272fe0334a95c4c1031e0e3723"} Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.387994 5272 scope.go:122] "RemoveContainer" containerID="8c9e0c5ef6b34bf1513995b31e1e80c5cb6385284211fdfa4ca1010daf18d4d2" Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.388732 5272 scope.go:122] "RemoveContainer" containerID="50f093cb23d42cd01d77b0712ae567e4df8135272fe0334a95c4c1031e0e3723" Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:23.389414 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895765-pxtmx_odh-ai-gateway-infra(09630af1-91d7-49d3-92b6-e3516312596c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" podUID="09630af1-91d7-49d3-92b6-e3516312596c" Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.404494 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerID="edfb742cec776e48d1c15e7fe6c746418d4730435d129840d5be78045e927999" exitCode=60 Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.404737 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" event={"ID":"1ffae27a-9b94-4cce-a0b4-b6946aa0612d","Type":"ContainerDied","Data":"edfb742cec776e48d1c15e7fe6c746418d4730435d129840d5be78045e927999"} Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.405660 5272 scope.go:122] "RemoveContainer" containerID="edfb742cec776e48d1c15e7fe6c746418d4730435d129840d5be78045e927999" Nov 03 22:45:23.406856 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:23.406606 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5_odh-ai-gateway-infra(1ffae27a-9b94-4cce-a0b4-b6946aa0612d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" Nov 03 22:45:23.416152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.409275 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" event={"ID":"a7db9322-4a97-46de-b7f6-70ea93c51df1","Type":"ContainerDied","Data":"cbc6777017b5cda8f3a6ce5ccfe00435f225267fecc0f485e1306490cd1aa6e8"} Nov 03 22:45:23.416152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.409384 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn" Nov 03 22:45:23.470537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.468139 5272 generic.go:356] "Generic (PLEG): container finished" podID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerID="308b8fad93c68fa06afe221fcd24c6c6d2437c727463549acc49719e532edb9d" exitCode=60 Nov 03 22:45:23.474329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.471907 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" event={"ID":"91e1c200-25f1-443d-9cb4-c4bcf283038e","Type":"ContainerDied","Data":"308b8fad93c68fa06afe221fcd24c6c6d2437c727463549acc49719e532edb9d"} Nov 03 22:45:23.477187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.476094 5272 scope.go:122] "RemoveContainer" containerID="308b8fad93c68fa06afe221fcd24c6c6d2437c727463549acc49719e532edb9d" Nov 03 22:45:23.477187 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:23.476642 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp_odh-ai-gateway-infra(91e1c200-25f1-443d-9cb4-c4bcf283038e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" Nov 03 22:45:23.661832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.651595 5272 scope.go:122] "RemoveContainer" containerID="1e6dc1e0edf1dedb9362146a7dddcead482d7d89da7ad58dc051160e66786129" Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.856500 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx"] Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.856563 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q"] Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.857681 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.857704 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.868326 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.868357 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.868680 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:23.872741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.868721 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:23.963852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.962007 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q"] Nov 03 22:45:23.963852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.962073 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p"] Nov 03 22:45:24.000828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:23.999951 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.031202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.022588 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfnr\" (UniqueName: \"kubernetes.io/projected/c875303f-3ae0-46ad-af10-bcfaea6232d5-kube-api-access-rxfnr\") pod \"payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.031202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.022856 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875303f-3ae0-46ad-af10-bcfaea6232d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.063741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.063710 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:24.063991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.063972 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:24.064339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.064326 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" containerName="cleanup" Nov 03 22:45:24.080866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.073714 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-65cac0\"" Nov 03 22:45:24.080866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.075148 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-65cac0-dockercfg-qctwz\"" Nov 03 22:45:24.113044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.113016 5272 scope.go:122] "RemoveContainer" containerID="c79328c0d3ccfc55788487ace7235d8f00a7d86567937b0b88e6bbbab305792a" Nov 03 22:45:24.131938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.129550 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfnr\" (UniqueName: \"kubernetes.io/projected/c875303f-3ae0-46ad-af10-bcfaea6232d5-kube-api-access-rxfnr\") pod \"payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.131938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.129870 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875303f-3ae0-46ad-af10-bcfaea6232d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.141526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.133594 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875303f-3ae0-46ad-af10-bcfaea6232d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.149246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.149123 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p"] Nov 03 22:45:24.149380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.149289 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp"] Nov 03 22:45:24.149460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.149324 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.180203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.172865 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn"] Nov 03 22:45:24.180203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.178377 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895765-vpcpn"] Nov 03 22:45:24.191225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.191100 5272 scope.go:122] "RemoveContainer" containerID="9e2464c5bd9bb5366a3a5e5f86debaeab3f5e65ca27070c82abd6897ccb3ec67" Nov 03 22:45:24.198458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.198359 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfnr\" (UniqueName: \"kubernetes.io/projected/c875303f-3ae0-46ad-af10-bcfaea6232d5-kube-api-access-rxfnr\") pod \"payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.200031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.199997 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:24.278674 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.278648 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:24.302368 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.302341 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:24.331022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.330998 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:24.334287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.334262 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f86136-0a0b-44ba-806c-df4a42538a3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f86136-0a0b-44ba-806c-df4a42538a3f-maas-api-service-ca\") pod \"80f86136-0a0b-44ba-806c-df4a42538a3f\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " Nov 03 22:45:24.334540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.334518 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f86136-0a0b-44ba-806c-df4a42538a3f-kube-api-access-sfdc7\" (UniqueName: \"kubernetes.io/projected/80f86136-0a0b-44ba-806c-df4a42538a3f-kube-api-access-sfdc7\") pod \"80f86136-0a0b-44ba-806c-df4a42538a3f\" (UID: \"80f86136-0a0b-44ba-806c-df4a42538a3f\") " Nov 03 22:45:24.334665 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.334644 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f86136-0a0b-44ba-806c-df4a42538a3f-maas-api-service-ca" pod "80f86136-0a0b-44ba-806c-df4a42538a3f" (UID: "80f86136-0a0b-44ba-806c-df4a42538a3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.335382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.335357 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c7dfd-8682-4669-9e6f-6c415fb5f353-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.335496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.335420 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrjn\" (UniqueName: \"kubernetes.io/projected/c60c7dfd-8682-4669-9e6f-6c415fb5f353-kube-api-access-ctrjn\") pod \"payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.335593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.335573 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f86136-0a0b-44ba-806c-df4a42538a3f-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.336607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.336585 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f86136-0a0b-44ba-806c-df4a42538a3f-kube-api-access-sfdc7" pod "80f86136-0a0b-44ba-806c-df4a42538a3f" (UID: "80f86136-0a0b-44ba-806c-df4a42538a3f"). InnerVolumeSpecName "kube-api-access-sfdc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.436571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.436491 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425f548c-b952-4b9f-8864-0f05377b538d-kube-api-access-78msx\" (UniqueName: \"kubernetes.io/projected/425f548c-b952-4b9f-8864-0f05377b538d-kube-api-access-78msx\") pod \"425f548c-b952-4b9f-8864-0f05377b538d\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " Nov 03 22:45:24.436571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.436543 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fbab71-53ea-449c-b01a-f1c6550a26c1-kube-api-access-7s8w9\" (UniqueName: \"kubernetes.io/projected/b9fbab71-53ea-449c-b01a-f1c6550a26c1-kube-api-access-7s8w9\") pod \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " Nov 03 22:45:24.437102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.436630 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41dd3f6-f48d-428e-af46-c8ba2ce71517-kube-api-access-f4msr\" (UniqueName: \"kubernetes.io/projected/f41dd3f6-f48d-428e-af46-c8ba2ce71517-kube-api-access-f4msr\") pod \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " Nov 03 22:45:24.437102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.436700 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fbab71-53ea-449c-b01a-f1c6550a26c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9fbab71-53ea-449c-b01a-f1c6550a26c1-maas-api-service-ca\") pod \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\" (UID: \"b9fbab71-53ea-449c-b01a-f1c6550a26c1\") " Nov 03 22:45:24.437102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.436723 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425f548c-b952-4b9f-8864-0f05377b538d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425f548c-b952-4b9f-8864-0f05377b538d-maas-api-service-ca\") pod \"425f548c-b952-4b9f-8864-0f05377b538d\" (UID: \"425f548c-b952-4b9f-8864-0f05377b538d\") " Nov 03 22:45:24.437102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.436773 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41dd3f6-f48d-428e-af46-c8ba2ce71517-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41dd3f6-f48d-428e-af46-c8ba2ce71517-maas-api-service-ca\") pod \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\" (UID: \"f41dd3f6-f48d-428e-af46-c8ba2ce71517\") " Nov 03 22:45:24.437294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437114 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fbab71-53ea-449c-b01a-f1c6550a26c1-maas-api-service-ca" pod "b9fbab71-53ea-449c-b01a-f1c6550a26c1" (UID: "b9fbab71-53ea-449c-b01a-f1c6550a26c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.437294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437175 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41dd3f6-f48d-428e-af46-c8ba2ce71517-maas-api-service-ca" pod "f41dd3f6-f48d-428e-af46-c8ba2ce71517" (UID: "f41dd3f6-f48d-428e-af46-c8ba2ce71517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.437538 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437516 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425f548c-b952-4b9f-8864-0f05377b538d-maas-api-service-ca" pod "425f548c-b952-4b9f-8864-0f05377b538d" (UID: "425f548c-b952-4b9f-8864-0f05377b538d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.437575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437519 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c7dfd-8682-4669-9e6f-6c415fb5f353-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.437632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437617 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrjn\" (UniqueName: \"kubernetes.io/projected/c60c7dfd-8682-4669-9e6f-6c415fb5f353-kube-api-access-ctrjn\") pod \"payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.437912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437892 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9fbab71-53ea-449c-b01a-f1c6550a26c1-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.437958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437918 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/425f548c-b952-4b9f-8864-0f05377b538d-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.437958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437933 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41dd3f6-f48d-428e-af46-c8ba2ce71517-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.437958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.437949 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdc7\" (UniqueName: \"kubernetes.io/projected/80f86136-0a0b-44ba-806c-df4a42538a3f-kube-api-access-sfdc7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.438245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.438227 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c7dfd-8682-4669-9e6f-6c415fb5f353-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.438937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.438917 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41dd3f6-f48d-428e-af46-c8ba2ce71517-kube-api-access-f4msr" pod "f41dd3f6-f48d-428e-af46-c8ba2ce71517" (UID: "f41dd3f6-f48d-428e-af46-c8ba2ce71517"). InnerVolumeSpecName "kube-api-access-f4msr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.439182 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.439163 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fbab71-53ea-449c-b01a-f1c6550a26c1-kube-api-access-7s8w9" pod "b9fbab71-53ea-449c-b01a-f1c6550a26c1" (UID: "b9fbab71-53ea-449c-b01a-f1c6550a26c1"). InnerVolumeSpecName "kube-api-access-7s8w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.439220 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.439193 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425f548c-b952-4b9f-8864-0f05377b538d-kube-api-access-78msx" pod "425f548c-b952-4b9f-8864-0f05377b538d" (UID: "425f548c-b952-4b9f-8864-0f05377b538d"). InnerVolumeSpecName "kube-api-access-78msx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.449571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.449551 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:24.495436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.495371 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrjn\" (UniqueName: \"kubernetes.io/projected/c60c7dfd-8682-4669-9e6f-6c415fb5f353-kube-api-access-ctrjn\") pod \"payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.503313 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:24.503277 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:45:24.539075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.539037 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78msx\" (UniqueName: \"kubernetes.io/projected/425f548c-b952-4b9f-8864-0f05377b538d-kube-api-access-78msx\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.539075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.539076 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8w9\" (UniqueName: \"kubernetes.io/projected/b9fbab71-53ea-449c-b01a-f1c6550a26c1-kube-api-access-7s8w9\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.539295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.539089 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4msr\" (UniqueName: \"kubernetes.io/projected/f41dd3f6-f48d-428e-af46-c8ba2ce71517-kube-api-access-f4msr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.577784 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.577532 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:24.687798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.687711 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" Nov 03 22:45:24.687798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.687731 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z" event={"ID":"425f548c-b952-4b9f-8864-0f05377b538d","Type":"ContainerDied","Data":"ca06418ffe44573e8dc2136f9270895010ca95e50d1632befe9a748f533185dc"} Nov 03 22:45:24.687798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.687818 5272 scope.go:122] "RemoveContainer" containerID="a9b2d5479929ffd4cc27652b2d08486e05f2d35fdfcfee5263c51c200b431a38" Nov 03 22:45:24.693361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.693336 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" event={"ID":"f41dd3f6-f48d-428e-af46-c8ba2ce71517","Type":"ContainerDied","Data":"51383edde7237d8f5ea3b0bf1afc4e0566255385fb33c307f75a91636f4209e3"} Nov 03 22:45:24.693490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.693344 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt" Nov 03 22:45:24.701388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.701254 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" event={"ID":"80f86136-0a0b-44ba-806c-df4a42538a3f","Type":"ContainerDied","Data":"c99db4c65fc8b459d51df1ff99d0c4b830283635b7c64bbb13f4a8922635667a"} Nov 03 22:45:24.701388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.701369 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k" Nov 03 22:45:24.703869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.703844 5272 scope.go:122] "RemoveContainer" containerID="98ed51bddba3c85c7cde6299d2f02bb3777d96ce41c9f7c63e5d504f1a1159df" Nov 03 22:45:24.725694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.725629 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" event={"ID":"b9fbab71-53ea-449c-b01a-f1c6550a26c1","Type":"ContainerDied","Data":"513ab33d4ad19f849cfe155bab325c7e17730e9d2897bb0cc33df9dcdcc48572"} Nov 03 22:45:24.725833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.725742 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb" Nov 03 22:45:24.726649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.726628 5272 scope.go:122] "RemoveContainer" containerID="677509f642ab398da610ac8eeea5868d98d7fcee87adfbd47b6a989b779c9a51" Nov 03 22:45:24.741076 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.741052 5272 scope.go:122] "RemoveContainer" containerID="079ed9e7d0ae5e645b8fe3b3c94dfbadc7d75c406010afe70f72dca3ffac5844" Nov 03 22:45:24.820084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.819931 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5"] Nov 03 22:45:24.839046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.839018 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q"] Nov 03 22:45:24.841062 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:45:24.841037 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc875303f_3ae0_46ad_af10_bcfaea6232d5.slice/crio-c5069577a867076c747bc579e25bbe29d68410c61f487487c7d55531a85f1075 WatchSource:0}: Error finding container c5069577a867076c747bc579e25bbe29d68410c61f487487c7d55531a85f1075: Status 404 returned error can't find the container with id c5069577a867076c747bc579e25bbe29d68410c61f487487c7d55531a85f1075 Nov 03 22:45:24.909642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.909580 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:24.944749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.944666 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09630af1-91d7-49d3-92b6-e3516312596c-kube-api-access-589cj\" (UniqueName: \"kubernetes.io/projected/09630af1-91d7-49d3-92b6-e3516312596c-kube-api-access-589cj\") pod \"09630af1-91d7-49d3-92b6-e3516312596c\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " Nov 03 22:45:24.944749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.944735 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09630af1-91d7-49d3-92b6-e3516312596c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09630af1-91d7-49d3-92b6-e3516312596c-maas-api-service-ca\") pod \"09630af1-91d7-49d3-92b6-e3516312596c\" (UID: \"09630af1-91d7-49d3-92b6-e3516312596c\") " Nov 03 22:45:24.945895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.945870 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09630af1-91d7-49d3-92b6-e3516312596c-maas-api-service-ca" pod "09630af1-91d7-49d3-92b6-e3516312596c" (UID: "09630af1-91d7-49d3-92b6-e3516312596c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.962150 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.962107 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09630af1-91d7-49d3-92b6-e3516312596c-kube-api-access-589cj" pod "09630af1-91d7-49d3-92b6-e3516312596c" (UID: "09630af1-91d7-49d3-92b6-e3516312596c"). InnerVolumeSpecName "kube-api-access-589cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.965614 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.965589 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:24.995473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:24.993682 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p"] Nov 03 22:45:25.048269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.048228 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e1c200-25f1-443d-9cb4-c4bcf283038e-kube-api-access-scskf\" (UniqueName: \"kubernetes.io/projected/91e1c200-25f1-443d-9cb4-c4bcf283038e-kube-api-access-scskf\") pod \"91e1c200-25f1-443d-9cb4-c4bcf283038e\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " Nov 03 22:45:25.049656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.049604 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e1c200-25f1-443d-9cb4-c4bcf283038e-maas-api-service-ca" pod "91e1c200-25f1-443d-9cb4-c4bcf283038e" (UID: "91e1c200-25f1-443d-9cb4-c4bcf283038e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:25.051899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.049788 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e1c200-25f1-443d-9cb4-c4bcf283038e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1c200-25f1-443d-9cb4-c4bcf283038e-maas-api-service-ca\") pod \"91e1c200-25f1-443d-9cb4-c4bcf283038e\" (UID: \"91e1c200-25f1-443d-9cb4-c4bcf283038e\") " Nov 03 22:45:25.070684 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.070594 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09630af1-91d7-49d3-92b6-e3516312596c-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.070684 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.070631 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1c200-25f1-443d-9cb4-c4bcf283038e-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.070684 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.070645 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589cj\" (UniqueName: \"kubernetes.io/projected/09630af1-91d7-49d3-92b6-e3516312596c-kube-api-access-589cj\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.109241 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.109011 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e1c200-25f1-443d-9cb4-c4bcf283038e-kube-api-access-scskf" pod "91e1c200-25f1-443d-9cb4-c4bcf283038e" (UID: "91e1c200-25f1-443d-9cb4-c4bcf283038e"). InnerVolumeSpecName "kube-api-access-scskf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:25.172393 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.172357 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scskf\" (UniqueName: \"kubernetes.io/projected/91e1c200-25f1-443d-9cb4-c4bcf283038e-kube-api-access-scskf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.198158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.198096 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:25.273627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.273593 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb"] Nov 03 22:45:25.273627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.273614 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-kube-api-access-2ss68\" (UniqueName: \"kubernetes.io/projected/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-kube-api-access-2ss68\") pod \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " Nov 03 22:45:25.273965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.273747 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-maas-api-service-ca\") pod \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\" (UID: \"1ffae27a-9b94-4cce-a0b4-b6946aa0612d\") " Nov 03 22:45:25.274129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.274109 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-maas-api-service-ca" pod "1ffae27a-9b94-4cce-a0b4-b6946aa0612d" (UID: "1ffae27a-9b94-4cce-a0b4-b6946aa0612d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:25.274348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.274320 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.277972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.277944 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-kube-api-access-2ss68" pod "1ffae27a-9b94-4cce-a0b4-b6946aa0612d" (UID: "1ffae27a-9b94-4cce-a0b4-b6946aa0612d"). InnerVolumeSpecName "kube-api-access-2ss68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:25.286693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.286668 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895765-98dhb"] Nov 03 22:45:25.375670 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.375641 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ss68\" (UniqueName: \"kubernetes.io/projected/1ffae27a-9b94-4cce-a0b4-b6946aa0612d-kube-api-access-2ss68\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:25.512164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.512065 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7db9322-4a97-46de-b7f6-70ea93c51df1" path="/var/lib/kubelet/pods/a7db9322-4a97-46de-b7f6-70ea93c51df1/volumes" Nov 03 22:45:25.514055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.514022 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" path="/var/lib/kubelet/pods/b9fbab71-53ea-449c-b01a-f1c6550a26c1/volumes" Nov 03 22:45:25.515191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.515172 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k"] Nov 03 22:45:25.515272 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.515214 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-68fbd1-29895765-tzj9k"] Nov 03 22:45:25.599606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.599571 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt"] Nov 03 22:45:25.606592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.606564 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895765-d4cqt"] Nov 03 22:45:25.737745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.737714 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" event={"ID":"c60c7dfd-8682-4669-9e6f-6c415fb5f353","Type":"ContainerStarted","Data":"9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d"} Nov 03 22:45:25.737962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.737765 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" event={"ID":"c60c7dfd-8682-4669-9e6f-6c415fb5f353","Type":"ContainerStarted","Data":"acbf6361d9572e3f4da5f39202b3800204621bc1e64b0e1ae41844f5db2e483f"} Nov 03 22:45:25.737962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.737827 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:25.739468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.739447 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" Nov 03 22:45:25.739586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.739465 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp" event={"ID":"91e1c200-25f1-443d-9cb4-c4bcf283038e","Type":"ContainerDied","Data":"33d77ce54fefa3159d521fc0a4a2f8ce5f57405c2a5d98aa278d953b7788185a"} Nov 03 22:45:25.739586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.739521 5272 scope.go:122] "RemoveContainer" containerID="308b8fad93c68fa06afe221fcd24c6c6d2437c727463549acc49719e532edb9d" Nov 03 22:45:25.743104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.743074 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" event={"ID":"c875303f-3ae0-46ad-af10-bcfaea6232d5","Type":"ContainerStarted","Data":"9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3"} Nov 03 22:45:25.743235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.743127 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" event={"ID":"c875303f-3ae0-46ad-af10-bcfaea6232d5","Type":"ContainerStarted","Data":"c5069577a867076c747bc579e25bbe29d68410c61f487487c7d55531a85f1075"} Nov 03 22:45:25.743235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.743198 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:25.748418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.748398 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" Nov 03 22:45:25.748537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.748426 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx" event={"ID":"09630af1-91d7-49d3-92b6-e3516312596c","Type":"ContainerDied","Data":"c5f90758b5f4b49e8e441d391f5dcacabf803925d0c8332f585bf6dd043f0125"} Nov 03 22:45:25.750623 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.750120 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" event={"ID":"1ffae27a-9b94-4cce-a0b4-b6946aa0612d","Type":"ContainerDied","Data":"b55a8c535fa37eabed278b95a9554e8ad05dc7750af4a4193cd282e5250563a0"} Nov 03 22:45:25.750623 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.750170 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5" Nov 03 22:45:25.752651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.752634 5272 scope.go:122] "RemoveContainer" containerID="50f093cb23d42cd01d77b0712ae567e4df8135272fe0334a95c4c1031e0e3723" Nov 03 22:45:25.763431 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.763413 5272 scope.go:122] "RemoveContainer" containerID="edfb742cec776e48d1c15e7fe6c746418d4730435d129840d5be78045e927999" Nov 03 22:45:25.816569 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.816538 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z"] Nov 03 22:45:25.821746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.821719 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895765-hlj2z"] Nov 03 22:45:25.924814 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.924774 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx"] Nov 03 22:45:25.930056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:25.930027 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895765-pxtmx"] Nov 03 22:45:26.196829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:26.195420 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" podStartSLOduration=3.195401178 podStartE2EDuration="3.195401178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:26.059279181 +0000 UTC m=+1306.681587497" watchObservedRunningTime="2026-11-03 22:45:26.195401178 +0000 UTC m=+1306.817709486" Nov 03 22:45:26.349215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:26.292907 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5"] Nov 03 22:45:26.349215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:26.310774 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-bb95c832-29895765-vdqm5"] Nov 03 22:45:26.444853 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:26.444783 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp"] Nov 03 22:45:26.453535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:26.453464 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895765-fpwvp"] Nov 03 22:45:26.548202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:26.548142 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" podStartSLOduration=3.54812267 podStartE2EDuration="3.54812267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:26.543471376 +0000 UTC m=+1307.165779704" watchObservedRunningTime="2026-11-03 22:45:26.54812267 +0000 UTC m=+1307.170430981" Nov 03 22:45:27.495851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:27.495816 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09630af1-91d7-49d3-92b6-e3516312596c" path="/var/lib/kubelet/pods/09630af1-91d7-49d3-92b6-e3516312596c/volumes" Nov 03 22:45:27.496314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:27.496299 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" path="/var/lib/kubelet/pods/1ffae27a-9b94-4cce-a0b4-b6946aa0612d/volumes" Nov 03 22:45:27.496712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:27.496699 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425f548c-b952-4b9f-8864-0f05377b538d" path="/var/lib/kubelet/pods/425f548c-b952-4b9f-8864-0f05377b538d/volumes" Nov 03 22:45:27.497486 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:27.497470 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" path="/var/lib/kubelet/pods/80f86136-0a0b-44ba-806c-df4a42538a3f/volumes" Nov 03 22:45:27.497895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:27.497882 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" path="/var/lib/kubelet/pods/91e1c200-25f1-443d-9cb4-c4bcf283038e/volumes" Nov 03 22:45:27.498286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:27.498274 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" path="/var/lib/kubelet/pods/f41dd3f6-f48d-428e-af46-c8ba2ce71517/volumes" Nov 03 22:45:29.079941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.079905 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx"] Nov 03 22:45:29.080619 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080604 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.080666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080620 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.080666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080632 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:45:29.080666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080638 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:45:29.080666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080645 5272 cpu_manager.go:414] "containerMap: removing container" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:45:29.080666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080650 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:45:29.080666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080660 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:45:29.080666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080664 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080672 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080676 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080687 5272 cpu_manager.go:414] "containerMap: removing container" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080692 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080701 5272 cpu_manager.go:414] "containerMap: removing container" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080706 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080717 5272 cpu_manager.go:414] "containerMap: removing container" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080721 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080728 5272 cpu_manager.go:414] "containerMap: removing container" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080732 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080738 5272 cpu_manager.go:414] "containerMap: removing container" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080742 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080751 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080755 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080761 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080765 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080773 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080777 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080795 5272 cpu_manager.go:414] "containerMap: removing container" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080810 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080817 5272 cpu_manager.go:414] "containerMap: removing container" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080822 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080828 5272 cpu_manager.go:414] "containerMap: removing container" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.080897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080834 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080954 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080961 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080969 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080976 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080981 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080986 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080991 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.080997 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081002 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081007 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081013 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081018 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081026 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081033 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081038 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081044 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:45:29.081646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.081049 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41dd3f6-f48d-428e-af46-c8ba2ce71517" containerName="cleanup" Nov 03 22:45:29.086331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.086315 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.138327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.138300 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx"] Nov 03 22:45:29.179171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.179139 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:45:29.179336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.179234 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:45:29.179336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.179269 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:45:29.181290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.181270 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:45:29.214628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.214595 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.214830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.214707 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.214830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.214811 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2td7r\" (UniqueName: \"kubernetes.io/projected/3b791918-ca5a-4173-9c5b-f3e22681f22d-kube-api-access-2td7r\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.316444 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.316405 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2td7r\" (UniqueName: \"kubernetes.io/projected/3b791918-ca5a-4173-9c5b-f3e22681f22d-kube-api-access-2td7r\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.316639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.316616 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.316735 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.316722 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.317318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.317286 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.319162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.319139 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.380037 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.379960 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2td7r\" (UniqueName: \"kubernetes.io/projected/3b791918-ca5a-4173-9c5b-f3e22681f22d-kube-api-access-2td7r\") pod \"maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.401114 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.401086 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:29.825972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.825788 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx"] Nov 03 22:45:29.829204 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:45:29.829178 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b791918_ca5a_4173_9c5b_f3e22681f22d.slice/crio-c9782097edabc35b1cfb7f5b1498bb36b8496ecdbccc1c19e447e99b9706cdef WatchSource:0}: Error finding container c9782097edabc35b1cfb7f5b1498bb36b8496ecdbccc1c19e447e99b9706cdef: Status 404 returned error can't find the container with id c9782097edabc35b1cfb7f5b1498bb36b8496ecdbccc1c19e447e99b9706cdef Nov 03 22:45:29.961423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:29.961338 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" event={"ID":"3b791918-ca5a-4173-9c5b-f3e22681f22d","Type":"ContainerStarted","Data":"c9782097edabc35b1cfb7f5b1498bb36b8496ecdbccc1c19e447e99b9706cdef"} Nov 03 22:45:30.983767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:30.983728 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" event={"ID":"3b791918-ca5a-4173-9c5b-f3e22681f22d","Type":"ContainerStarted","Data":"265391b72ef7afe97b4ee398cda8b61a47326bc82ea482bb89950f8b680c236c"} Nov 03 22:45:31.993858 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:31.993823 5272 generic.go:356] "Generic (PLEG): container finished" podID="3b791918-ca5a-4173-9c5b-f3e22681f22d" containerID="265391b72ef7afe97b4ee398cda8b61a47326bc82ea482bb89950f8b680c236c" exitCode=0 Nov 03 22:45:31.994512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:31.994489 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" event={"ID":"3b791918-ca5a-4173-9c5b-f3e22681f22d","Type":"ContainerDied","Data":"265391b72ef7afe97b4ee398cda8b61a47326bc82ea482bb89950f8b680c236c"} Nov 03 22:45:33.195668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.195642 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:33.292382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.292351 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-openshift-ca\") pod \"3b791918-ca5a-4173-9c5b-f3e22681f22d\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " Nov 03 22:45:33.292574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.292403 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b791918-ca5a-4173-9c5b-f3e22681f22d-kube-api-access-2td7r\" (UniqueName: \"kubernetes.io/projected/3b791918-ca5a-4173-9c5b-f3e22681f22d-kube-api-access-2td7r\") pod \"3b791918-ca5a-4173-9c5b-f3e22681f22d\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " Nov 03 22:45:33.292574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.292543 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-xks-ca\") pod \"3b791918-ca5a-4173-9c5b-f3e22681f22d\" (UID: \"3b791918-ca5a-4173-9c5b-f3e22681f22d\") " Nov 03 22:45:33.292855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.292790 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-openshift-ca" pod "3b791918-ca5a-4173-9c5b-f3e22681f22d" (UID: "3b791918-ca5a-4173-9c5b-f3e22681f22d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:33.293187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.293157 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:33.294708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.294679 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-xks-ca" pod "3b791918-ca5a-4173-9c5b-f3e22681f22d" (UID: "3b791918-ca5a-4173-9c5b-f3e22681f22d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:33.294708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.294694 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b791918-ca5a-4173-9c5b-f3e22681f22d-kube-api-access-2td7r" pod "3b791918-ca5a-4173-9c5b-f3e22681f22d" (UID: "3b791918-ca5a-4173-9c5b-f3e22681f22d"). InnerVolumeSpecName "kube-api-access-2td7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:33.394622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.394586 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b791918-ca5a-4173-9c5b-f3e22681f22d-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:33.394622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:33.394619 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2td7r\" (UniqueName: \"kubernetes.io/projected/3b791918-ca5a-4173-9c5b-f3e22681f22d-kube-api-access-2td7r\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:34.013348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:34.013318 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" Nov 03 22:45:34.013348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:34.013332 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx" event={"ID":"3b791918-ca5a-4173-9c5b-f3e22681f22d","Type":"ContainerDied","Data":"c9782097edabc35b1cfb7f5b1498bb36b8496ecdbccc1c19e447e99b9706cdef"} Nov 03 22:45:34.013564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:34.013389 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9782097edabc35b1cfb7f5b1498bb36b8496ecdbccc1c19e447e99b9706cdef" Nov 03 22:45:34.852041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:34.851991 5272 patch_prober.go:29] interesting pod/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:34.852574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:34.852086 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:35.053507 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:35.053463 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp"] Nov 03 22:45:35.094162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:35.094113 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c62007c-qbggp"] Nov 03 22:45:35.495697 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:35.495659 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8fcc5b-a162-440f-b537-77bf5f566a85" path="/var/lib/kubelet/pods/8f8fcc5b-a162-440f-b537-77bf5f566a85/volumes" Nov 03 22:45:36.838943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:36.838914 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 22:45:36.839380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:36.839330 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 22:45:37.046646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:37.046612 5272 generic.go:356] "Generic (PLEG): container finished" podID="77f08872-278e-4f08-9a4d-11e1e223321a" containerID="7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60" exitCode=0 Nov 03 22:45:37.046875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:37.046690 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" event={"ID":"77f08872-278e-4f08-9a4d-11e1e223321a","Type":"ContainerDied","Data":"7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60"} Nov 03 22:45:37.047180 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:37.047164 5272 scope.go:122] "RemoveContainer" containerID="7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60" Nov 03 22:45:37.048843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:37.048792 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:37.048939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:37.048878 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:37.048939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:37.048909 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:38.075280 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:38.075240 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" event={"ID":"77f08872-278e-4f08-9a4d-11e1e223321a","Type":"ContainerStarted","Data":"990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51"} Nov 03 22:45:38.814192 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:38.814160 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:45:38.814502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:38.814478 5272 patch_prober.go:29] interesting pod/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:45:38.814605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:38.814532 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 22:45:39.815095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:39.815054 5272 patch_prober.go:29] interesting pod/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:45:39.815618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:39.815122 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 22:45:40.814215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:40.814183 5272 patch_prober.go:29] interesting pod/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:45:40.814406 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:40.814235 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 22:45:41.814715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:41.814681 5272 patch_prober.go:29] interesting pod/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:45:41.815174 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:41.814766 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 22:45:41.975282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:41.975242 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn"] Nov 03 22:45:42.102124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:42.102041 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795"] Nov 03 22:45:42.814610 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:42.814574 5272 patch_prober.go:29] interesting pod/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 22:45:42.814833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:42.814627 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 22:45:43.834161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:43.834122 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:45:43.838164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:43.838138 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:45:43.843875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:43.843851 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:45:46.976356 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:46.976301 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" podUID="1bcbd3c5-4447-4a79-b5f4-7140966c958d" containerName="payload-processing" containerID="cri-o://71254ed5fd0d80a1d7f361f844a11a879f33da7df38e86a369cdfa90958ca302" gracePeriod=25 Nov 03 22:45:47.102815 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.102741 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" podUID="d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" containerName="payload-pre-processing" containerID="cri-o://887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1" gracePeriod=25 Nov 03 22:45:47.172657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.172618 5272 generic.go:356] "Generic (PLEG): container finished" podID="1bcbd3c5-4447-4a79-b5f4-7140966c958d" containerID="71254ed5fd0d80a1d7f361f844a11a879f33da7df38e86a369cdfa90958ca302" exitCode=0 Nov 03 22:45:47.173140 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.173115 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" event={"ID":"1bcbd3c5-4447-4a79-b5f4-7140966c958d","Type":"ContainerDied","Data":"71254ed5fd0d80a1d7f361f844a11a879f33da7df38e86a369cdfa90958ca302"} Nov 03 22:45:47.279363 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.279321 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:45:47.317706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.317673 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcbd3c5-4447-4a79-b5f4-7140966c958d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcbd3c5-4447-4a79-b5f4-7140966c958d-plugins-config-volume\") pod \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " Nov 03 22:45:47.317914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.317897 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcbd3c5-4447-4a79-b5f4-7140966c958d-kube-api-access-8hhh2\" (UniqueName: \"kubernetes.io/projected/1bcbd3c5-4447-4a79-b5f4-7140966c958d-kube-api-access-8hhh2\") pod \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\" (UID: \"1bcbd3c5-4447-4a79-b5f4-7140966c958d\") " Nov 03 22:45:47.318139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.318108 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcbd3c5-4447-4a79-b5f4-7140966c958d-plugins-config-volume" pod "1bcbd3c5-4447-4a79-b5f4-7140966c958d" (UID: "1bcbd3c5-4447-4a79-b5f4-7140966c958d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:47.318591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.318569 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcbd3c5-4447-4a79-b5f4-7140966c958d-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:47.320207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.320180 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcbd3c5-4447-4a79-b5f4-7140966c958d-kube-api-access-8hhh2" pod "1bcbd3c5-4447-4a79-b5f4-7140966c958d" (UID: "1bcbd3c5-4447-4a79-b5f4-7140966c958d"). InnerVolumeSpecName "kube-api-access-8hhh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:47.405285 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.405256 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:45:47.419826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.419772 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhh2\" (UniqueName: \"kubernetes.io/projected/1bcbd3c5-4447-4a79-b5f4-7140966c958d-kube-api-access-8hhh2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:47.521040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.521007 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-plugins-config-volume\") pod \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " Nov 03 22:45:47.524668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.524638 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-kube-api-access-r27vf\" (UniqueName: \"kubernetes.io/projected/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-kube-api-access-r27vf\") pod \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\" (UID: \"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5\") " Nov 03 22:45:47.525569 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.525541 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-plugins-config-volume" pod "d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" (UID: "d0970e14-cd44-44db-b63b-7d5c6fd7d3d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:47.528925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.528888 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-kube-api-access-r27vf" pod "d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" (UID: "d0970e14-cd44-44db-b63b-7d5c6fd7d3d5"). InnerVolumeSpecName "kube-api-access-r27vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:47.626768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.626734 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27vf\" (UniqueName: \"kubernetes.io/projected/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-kube-api-access-r27vf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:47.626768 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:47.626764 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:48.183530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.183496 5272 generic.go:356] "Generic (PLEG): container finished" podID="d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" containerID="887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1" exitCode=0 Nov 03 22:45:48.184019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.183542 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" Nov 03 22:45:48.184019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.183578 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" event={"ID":"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5","Type":"ContainerDied","Data":"887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1"} Nov 03 22:45:48.184019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.183640 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795" event={"ID":"d0970e14-cd44-44db-b63b-7d5c6fd7d3d5","Type":"ContainerDied","Data":"fce988741f4c898381b08771bb32ccc39cbdd5c2eb5f0fe46ad9786b1fed36e7"} Nov 03 22:45:48.184019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.183672 5272 scope.go:122] "RemoveContainer" containerID="887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1" Nov 03 22:45:48.185458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.185439 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" Nov 03 22:45:48.185458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.185451 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn" event={"ID":"1bcbd3c5-4447-4a79-b5f4-7140966c958d","Type":"ContainerDied","Data":"76a7c896dae4845fd0ece7b7b5ee00ddf3be8c4a543116e1a581e6513ade7fc5"} Nov 03 22:45:48.196788 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.196772 5272 scope.go:122] "RemoveContainer" containerID="887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1" Nov 03 22:45:48.197131 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:48.197109 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1\": container with ID starting with 887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1 not found: ID does not exist" containerID="887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1" Nov 03 22:45:48.197194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.197140 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1"} err="failed to get container status \"887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1\": rpc error: code = NotFound desc = could not find container \"887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1\": container with ID starting with 887f54c4a86aced33523b582a033d3b5e0584b97ef19ce41cfffa67fad45f4e1 not found: ID does not exist" Nov 03 22:45:48.197194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.197158 5272 scope.go:122] "RemoveContainer" containerID="71254ed5fd0d80a1d7f361f844a11a879f33da7df38e86a369cdfa90958ca302" Nov 03 22:45:48.340236 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.340199 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn"] Nov 03 22:45:48.386350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.386314 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-027a6fbe-d6957d7c4-rprgn"] Nov 03 22:45:48.446921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.446885 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795"] Nov 03 22:45:48.451631 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:48.451603 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-027a6fbe-658459cfb7-qg795"] Nov 03 22:45:49.047249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:49.047209 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf"] Nov 03 22:45:49.093673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:49.093634 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42356dd9-cjwwf"] Nov 03 22:45:49.495180 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:49.495101 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcbd3c5-4447-4a79-b5f4-7140966c958d" path="/var/lib/kubelet/pods/1bcbd3c5-4447-4a79-b5f4-7140966c958d/volumes" Nov 03 22:45:49.495623 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:49.495558 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1fd1c4e-3d51-4757-8853-2a0370512b0f" path="/var/lib/kubelet/pods/a1fd1c4e-3d51-4757-8853-2a0370512b0f/volumes" Nov 03 22:45:49.496004 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:45:49.495985 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" path="/var/lib/kubelet/pods/d0970e14-cd44-44db-b63b-7d5c6fd7d3d5/volumes" Nov 03 22:45:54.502522 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:45:54.502479 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:46:00.435516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:00.435473 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:46:13.691253 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.691212 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv"] Nov 03 22:46:13.691967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.691950 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" containerName="payload-pre-processing" Nov 03 22:46:13.692035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.691968 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" containerName="payload-pre-processing" Nov 03 22:46:13.692035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.691985 5272 cpu_manager.go:414] "containerMap: removing container" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:46:13.692035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.691994 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:46:13.692035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692007 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:46:13.692035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692014 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:46:13.692035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692036 5272 cpu_manager.go:414] "containerMap: removing container" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692042 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692051 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692055 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffae27a-9b94-4cce-a0b4-b6946aa0612d" containerName="cleanup" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692067 5272 cpu_manager.go:414] "containerMap: removing container" podUID="3b791918-ca5a-4173-9c5b-f3e22681f22d" containerName="revoke-keys" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692072 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b791918-ca5a-4173-9c5b-f3e22681f22d" containerName="revoke-keys" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692080 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1bcbd3c5-4447-4a79-b5f4-7140966c958d" containerName="payload-processing" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692084 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcbd3c5-4447-4a79-b5f4-7140966c958d" containerName="payload-processing" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692096 5272 cpu_manager.go:414] "containerMap: removing container" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692101 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09630af1-91d7-49d3-92b6-e3516312596c" containerName="cleanup" Nov 03 22:46:13.692219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692219 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b791918-ca5a-4173-9c5b-f3e22681f22d" containerName="revoke-keys" Nov 03 22:46:13.692516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692228 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcbd3c5-4447-4a79-b5f4-7140966c958d" containerName="payload-processing" Nov 03 22:46:13.692516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692244 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fbab71-53ea-449c-b01a-f1c6550a26c1" containerName="cleanup" Nov 03 22:46:13.692516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692256 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0970e14-cd44-44db-b63b-7d5c6fd7d3d5" containerName="payload-pre-processing" Nov 03 22:46:13.692516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692264 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="425f548c-b952-4b9f-8864-0f05377b538d" containerName="cleanup" Nov 03 22:46:13.692516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692270 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e1c200-25f1-443d-9cb4-c4bcf283038e" containerName="cleanup" Nov 03 22:46:13.692516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.692275 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f86136-0a0b-44ba-806c-df4a42538a3f" containerName="cleanup" Nov 03 22:46:13.695418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.695400 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:13.749475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.749443 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv"] Nov 03 22:46:13.790109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.790064 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:46:13.790109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.790064 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:13.790109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.790070 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:46:13.790872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.790844 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:46:13.843333 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.843300 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh"] Nov 03 22:46:13.843660 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.843595 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" containerID="cri-o://712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4" gracePeriod=30 Nov 03 22:46:13.869754 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.869714 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:13.870024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.869994 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnnf\" (UniqueName: \"kubernetes.io/projected/57204af1-6a8c-4741-bb82-9e6d178378d4-kube-api-access-ncnnf\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:13.870099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.870081 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:13.990889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.990306 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnnf\" (UniqueName: \"kubernetes.io/projected/57204af1-6a8c-4741-bb82-9e6d178378d4-kube-api-access-ncnnf\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:13.990889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.990402 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:13.990889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:13.990599 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:14.020833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:14.006288 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:14.027141 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:14.024450 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:14.091826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:14.090665 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnnf\" (UniqueName: \"kubernetes.io/projected/57204af1-6a8c-4741-bb82-9e6d178378d4-kube-api-access-ncnnf\") pod \"maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:14.311012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:14.310971 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:14.695795 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:14.695719 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv"] Nov 03 22:46:14.699781 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:46:14.699750 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57204af1_6a8c_4741_bb82_9e6d178378d4.slice/crio-19c3557c9bae3ca41f036138bbff681011e352aa86eb076bd398f45a5de40814 WatchSource:0}: Error finding container 19c3557c9bae3ca41f036138bbff681011e352aa86eb076bd398f45a5de40814: Status 404 returned error can't find the container with id 19c3557c9bae3ca41f036138bbff681011e352aa86eb076bd398f45a5de40814 Nov 03 22:46:15.015158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.015073 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6"] Nov 03 22:46:15.021919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.021887 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.046905 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.046867 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6"] Nov 03 22:46:15.075029 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.075006 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v"] Nov 03 22:46:15.078938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.078919 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132486 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpp5\" (UniqueName: \"kubernetes.io/projected/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-kube-api-access-sfpp5\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132545 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132590 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-envoy\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132616 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132650 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-token\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132673 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03aadb43-1ccb-4185-af97-9b69e3ee460b-istiod-ca-cert\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132782 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-data\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.132864 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-socket\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.133249 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-podinfo\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.133319 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-certs\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.133351 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-credential-socket\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.133827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.133503 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkmr\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-kube-api-access-5dkmr\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.316499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.316455 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v"] Nov 03 22:46:15.321063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.320963 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-podinfo\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.321368 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.321335 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-certs\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.321510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.321495 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-credential-socket\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.321719 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.321697 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkmr\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-kube-api-access-5dkmr\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.321907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.321888 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpp5\" (UniqueName: \"kubernetes.io/projected/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-kube-api-access-sfpp5\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.322049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.322032 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.322249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.322231 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-envoy\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.322360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.322347 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.322470 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.322458 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-token\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.322558 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.322545 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03aadb43-1ccb-4185-af97-9b69e3ee460b-istiod-ca-cert\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.322738 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.322726 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-data\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.322921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.322905 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-socket\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.323599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.323577 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-socket\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.327161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.327137 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-podinfo\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.327873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.327841 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-certs\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.328259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.328237 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-credential-socket\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.372824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.367299 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.372824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.367958 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03aadb43-1ccb-4185-af97-9b69e3ee460b-istiod-ca-cert\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.386603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.386190 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-data\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.411128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.410817 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.463843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.454180 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-envoy\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.484855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.470277 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e7a91ca4-openshift-default-dockercfg-h5xnq\"" Nov 03 22:46:15.587350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.586918 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkmr\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-kube-api-access-5dkmr\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.602836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.601958 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-token\") pod \"e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.602836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.602639 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" event={"ID":"57204af1-6a8c-4741-bb82-9e6d178378d4","Type":"ContainerStarted","Data":"8d64e11c95b6bc1ce220af156fcdc75c7d3928861e6a32f9db9dc8ad67329773"} Nov 03 22:46:15.602836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.602706 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" event={"ID":"57204af1-6a8c-4741-bb82-9e6d178378d4","Type":"ContainerStarted","Data":"19c3557c9bae3ca41f036138bbff681011e352aa86eb076bd398f45a5de40814"} Nov 03 22:46:15.655860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.655681 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpp5\" (UniqueName: \"kubernetes.io/projected/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-kube-api-access-sfpp5\") pod \"maas-api-revoke-keys-e2e-ait-c6905007-5gtd6\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.679568 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.679510 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:15.716836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.700605 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:15.734639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:15.734585 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" podStartSLOduration=2.73456136 podStartE2EDuration="2.73456136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:15.71948062 +0000 UTC m=+1356.341788941" watchObservedRunningTime="2026-11-03 22:46:15.73456136 +0000 UTC m=+1356.356869669" Nov 03 22:46:16.052250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.052215 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2"] Nov 03 22:46:16.057189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.057163 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4f98ab-4zjv2"] Nov 03 22:46:16.103289 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.103255 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6"] Nov 03 22:46:16.147445 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:46:16.147415 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdafdd1_8b0f_4ad1_9985_da859d5f3e5d.slice/crio-86cfd9ea1ff0885cd810208cc161cefa9f777d0301a53192d522ac6505b24425 WatchSource:0}: Error finding container 86cfd9ea1ff0885cd810208cc161cefa9f777d0301a53192d522ac6505b24425: Status 404 returned error can't find the container with id 86cfd9ea1ff0885cd810208cc161cefa9f777d0301a53192d522ac6505b24425 Nov 03 22:46:16.164121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.164089 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v"] Nov 03 22:46:16.172906 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:46:16.172865 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03aadb43_1ccb_4185_af97_9b69e3ee460b.slice/crio-eced1e2bf4d80056901bf1355aead38c8d2f7bc6906704e59f6526d41f7f3048 WatchSource:0}: Error finding container eced1e2bf4d80056901bf1355aead38c8d2f7bc6906704e59f6526d41f7f3048: Status 404 returned error can't find the container with id eced1e2bf4d80056901bf1355aead38c8d2f7bc6906704e59f6526d41f7f3048 Nov 03 22:46:16.176407 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.176353 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:46:16.176524 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.176505 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:46:16.176663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.176639 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:46:16.613818 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.613697 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" event={"ID":"03aadb43-1ccb-4185-af97-9b69e3ee460b","Type":"ContainerStarted","Data":"a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa"} Nov 03 22:46:16.613818 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.613769 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" event={"ID":"03aadb43-1ccb-4185-af97-9b69e3ee460b","Type":"ContainerStarted","Data":"eced1e2bf4d80056901bf1355aead38c8d2f7bc6906704e59f6526d41f7f3048"} Nov 03 22:46:16.616127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.616096 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" event={"ID":"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d","Type":"ContainerStarted","Data":"7e05145d71278764fc4733b29ba41d6b0b89346d61d0ceee17175d6d2a125202"} Nov 03 22:46:16.616270 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.616141 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" event={"ID":"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d","Type":"ContainerStarted","Data":"86cfd9ea1ff0885cd810208cc161cefa9f777d0301a53192d522ac6505b24425"} Nov 03 22:46:16.619080 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.619050 5272 generic.go:356] "Generic (PLEG): container finished" podID="57204af1-6a8c-4741-bb82-9e6d178378d4" containerID="8d64e11c95b6bc1ce220af156fcdc75c7d3928861e6a32f9db9dc8ad67329773" exitCode=0 Nov 03 22:46:16.619191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.619093 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" event={"ID":"57204af1-6a8c-4741-bb82-9e6d178378d4","Type":"ContainerDied","Data":"8d64e11c95b6bc1ce220af156fcdc75c7d3928861e6a32f9db9dc8ad67329773"} Nov 03 22:46:16.721570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.721528 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:16.731360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:16.731294 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podStartSLOduration=1.7312753650000001 podStartE2EDuration="1.731275365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:16.726610133 +0000 UTC m=+1357.348918446" watchObservedRunningTime="2026-11-03 22:46:16.731275365 +0000 UTC m=+1357.353583675" Nov 03 22:46:17.204830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.204779 5272 patch_prober.go:29] interesting pod/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:46:17.205019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.204871 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:46:17.495730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.495650 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5406d040-614e-4940-89e4-1220b6dcf2dc" path="/var/lib/kubelet/pods/5406d040-614e-4940-89e4-1220b6dcf2dc/volumes" Nov 03 22:46:17.629341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.629305 5272 generic.go:356] "Generic (PLEG): container finished" podID="0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" containerID="7e05145d71278764fc4733b29ba41d6b0b89346d61d0ceee17175d6d2a125202" exitCode=0 Nov 03 22:46:17.629527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.629390 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" event={"ID":"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d","Type":"ContainerDied","Data":"7e05145d71278764fc4733b29ba41d6b0b89346d61d0ceee17175d6d2a125202"} Nov 03 22:46:17.729432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.729400 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:17.814736 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.814703 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:17.903826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.901718 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57204af1-6a8c-4741-bb82-9e6d178378d4-kube-api-access-ncnnf\" (UniqueName: \"kubernetes.io/projected/57204af1-6a8c-4741-bb82-9e6d178378d4-kube-api-access-ncnnf\") pod \"57204af1-6a8c-4741-bb82-9e6d178378d4\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " Nov 03 22:46:17.903826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.902401 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-xks-ca\") pod \"57204af1-6a8c-4741-bb82-9e6d178378d4\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " Nov 03 22:46:17.903826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.902782 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-openshift-ca\") pod \"57204af1-6a8c-4741-bb82-9e6d178378d4\" (UID: \"57204af1-6a8c-4741-bb82-9e6d178378d4\") " Nov 03 22:46:17.903826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.903256 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-openshift-ca" pod "57204af1-6a8c-4741-bb82-9e6d178378d4" (UID: "57204af1-6a8c-4741-bb82-9e6d178378d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:17.903826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.903669 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:17.907086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.907056 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57204af1-6a8c-4741-bb82-9e6d178378d4-kube-api-access-ncnnf" pod "57204af1-6a8c-4741-bb82-9e6d178378d4" (UID: "57204af1-6a8c-4741-bb82-9e6d178378d4"). InnerVolumeSpecName "kube-api-access-ncnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:17.909067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:17.909030 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-xks-ca" pod "57204af1-6a8c-4741-bb82-9e6d178378d4" (UID: "57204af1-6a8c-4741-bb82-9e6d178378d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:18.014871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:18.014784 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnnf\" (UniqueName: \"kubernetes.io/projected/57204af1-6a8c-4741-bb82-9e6d178378d4-kube-api-access-ncnnf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:18.014871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:18.014843 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57204af1-6a8c-4741-bb82-9e6d178378d4-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:18.641036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:18.640997 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" Nov 03 22:46:18.641036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:18.640997 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv" event={"ID":"57204af1-6a8c-4741-bb82-9e6d178378d4","Type":"ContainerDied","Data":"19c3557c9bae3ca41f036138bbff681011e352aa86eb076bd398f45a5de40814"} Nov 03 22:46:18.641259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:18.641089 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c3557c9bae3ca41f036138bbff681011e352aa86eb076bd398f45a5de40814" Nov 03 22:46:18.641425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:18.641400 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:18.642378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:18.642359 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:46:19.105043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.105010 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:19.181439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.181373 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:46:19.198874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.198793 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-kube-api-access-sfpp5\" (UniqueName: \"kubernetes.io/projected/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-kube-api-access-sfpp5\") pod \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " Nov 03 22:46:19.199060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.198944 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-xks-ca\") pod \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " Nov 03 22:46:19.199060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.198988 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-openshift-ca\") pod \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\" (UID: \"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d\") " Nov 03 22:46:19.199400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.199372 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-openshift-ca" pod "0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" (UID: "0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:19.199875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.199850 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.201422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.201397 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-kube-api-access-sfpp5" pod "0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" (UID: "0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d"). InnerVolumeSpecName "kube-api-access-sfpp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:19.201611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.201597 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-xks-ca" pod "0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" (UID: "0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:19.301050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301017 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-kube-api-access-2wqxh\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-kube-api-access-2wqxh\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301070 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-podinfo\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301100 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istiod-ca-cert\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301219 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-certs\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301330 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-data\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301369 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-token\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-token\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301413 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-socket\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301524 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-envoy\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301577 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-credential-socket\") pod \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\" (UID: \"c33bd25b-b162-47bd-aae3-e523f3ee31a7\") " Nov 03 22:46:19.301616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301519 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-certs" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:19.301722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301557 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-data" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:19.301722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301593 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istiod-ca-cert" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:19.301887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301715 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-socket" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:19.301980 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.301960 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-credential-socket" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:19.302327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.302311 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.302386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.302330 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.302386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.302346 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpp5\" (UniqueName: \"kubernetes.io/projected/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-kube-api-access-sfpp5\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.302386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.302357 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.302386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.302367 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.302386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.302376 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.302386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.302384 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.303782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.303764 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-podinfo" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:46:19.303884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.303784 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-kube-api-access-2wqxh" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "kube-api-access-2wqxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:19.304141 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.304125 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-envoy" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:19.304201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.304165 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-token" pod "c33bd25b-b162-47bd-aae3-e523f3ee31a7" (UID: "c33bd25b-b162-47bd-aae3-e523f3ee31a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:19.403904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.403862 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.403904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.403898 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.403904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.403908 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqxh\" (UniqueName: \"kubernetes.io/projected/c33bd25b-b162-47bd-aae3-e523f3ee31a7-kube-api-access-2wqxh\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.404160 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.403921 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bd25b-b162-47bd-aae3-e523f3ee31a7-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:19.691116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.690851 5272 generic.go:356] "Generic (PLEG): container finished" podID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerID="712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4" exitCode=0 Nov 03 22:46:19.703561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.703510 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" Nov 03 22:46:19.703968 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.703928 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" event={"ID":"c33bd25b-b162-47bd-aae3-e523f3ee31a7","Type":"ContainerDied","Data":"712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4"} Nov 03 22:46:19.704104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.704001 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh" event={"ID":"c33bd25b-b162-47bd-aae3-e523f3ee31a7","Type":"ContainerDied","Data":"4e1b68a5496222b2051b8c844b83a9a4c5150cf29098b5ed235ef1cbe5894a25"} Nov 03 22:46:19.704104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.704028 5272 scope.go:122] "RemoveContainer" containerID="712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4" Nov 03 22:46:19.706930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.706907 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" Nov 03 22:46:19.707133 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.707019 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6" event={"ID":"0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d","Type":"ContainerDied","Data":"86cfd9ea1ff0885cd810208cc161cefa9f777d0301a53192d522ac6505b24425"} Nov 03 22:46:19.707209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.707179 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86cfd9ea1ff0885cd810208cc161cefa9f777d0301a53192d522ac6505b24425" Nov 03 22:46:19.717378 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.717354 5272 scope.go:122] "RemoveContainer" containerID="1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f" Nov 03 22:46:19.728598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.728577 5272 scope.go:122] "RemoveContainer" containerID="712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4" Nov 03 22:46:19.728931 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:19.728909 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4\": container with ID starting with 712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4 not found: ID does not exist" containerID="712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4" Nov 03 22:46:19.729036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.728945 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4"} err="failed to get container status \"712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4\": rpc error: code = NotFound desc = could not find container \"712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4\": container with ID starting with 712651bb953902993f305f8a6096e57b99cf373f91d2d048a0d40d200f1847a4 not found: ID does not exist" Nov 03 22:46:19.729036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.728970 5272 scope.go:122] "RemoveContainer" containerID="1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f" Nov 03 22:46:19.729261 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:19.729242 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f\": container with ID starting with 1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f not found: ID does not exist" containerID="1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f" Nov 03 22:46:19.729300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.729271 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f"} err="failed to get container status \"1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f\": rpc error: code = NotFound desc = could not find container \"1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f\": container with ID starting with 1da9c6f87627deb8c4b9736bdab0105209ef755f5be6e2c5f658431d7b47d73f not found: ID does not exist" Nov 03 22:46:19.861624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.861590 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh"] Nov 03 22:46:19.907766 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:19.907729 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bb95c832-openshift-default-997798bd5-x5tlh"] Nov 03 22:46:21.557829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:21.555324 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" path="/var/lib/kubelet/pods/c33bd25b-b162-47bd-aae3-e523f3ee31a7/volumes" Nov 03 22:46:24.276902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.276856 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2"] Nov 03 22:46:24.280357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.280310 5272 cpu_manager.go:414] "containerMap: removing container" podUID="57204af1-6a8c-4741-bb82-9e6d178378d4" containerName="revoke-keys" Nov 03 22:46:24.280599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.280577 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57204af1-6a8c-4741-bb82-9e6d178378d4" containerName="revoke-keys" Nov 03 22:46:24.280767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.280752 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" Nov 03 22:46:24.281031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.281015 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" Nov 03 22:46:24.281208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.281184 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" containerName="revoke-keys" Nov 03 22:46:24.281325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.281312 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" containerName="revoke-keys" Nov 03 22:46:24.281458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.281433 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" Nov 03 22:46:24.281553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.281539 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" Nov 03 22:46:24.282073 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.282045 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="57204af1-6a8c-4741-bb82-9e6d178378d4" containerName="revoke-keys" Nov 03 22:46:24.282263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.282200 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" Nov 03 22:46:24.282388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.282373 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" containerName="revoke-keys" Nov 03 22:46:24.299198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.299139 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:24.310869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.310367 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01619c55-1af7-4068-8cbf-8ffd9da02445-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:24.310869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.310638 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbtw\" (UniqueName: \"kubernetes.io/projected/01619c55-1af7-4068-8cbf-8ffd9da02445-kube-api-access-sqbtw\") pod \"payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:24.320842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.318466 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2"] Nov 03 22:46:24.395341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.395301 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e7a91ca4-dockercfg-9j6fq\"" Nov 03 22:46:24.395341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.395317 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e7a91ca4\"" Nov 03 22:46:24.412025 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.411836 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01619c55-1af7-4068-8cbf-8ffd9da02445-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:24.412204 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.412053 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbtw\" (UniqueName: \"kubernetes.io/projected/01619c55-1af7-4068-8cbf-8ffd9da02445-kube-api-access-sqbtw\") pod \"payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:24.413055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.413034 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01619c55-1af7-4068-8cbf-8ffd9da02445-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:24.472556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.472523 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbtw\" (UniqueName: \"kubernetes.io/projected/01619c55-1af7-4068-8cbf-8ffd9da02445-kube-api-access-sqbtw\") pod \"payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:24.501758 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:24.501724 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:46:24.627366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:24.627334 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:25.015290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:25.015267 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2"] Nov 03 22:46:25.018238 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:46:25.018208 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01619c55_1af7_4068_8cbf_8ffd9da02445.slice/crio-dbd095bd29437c58261c9f17a9f2c67bfc9ee663fd9a1834698ac61fe9153462 WatchSource:0}: Error finding container dbd095bd29437c58261c9f17a9f2c67bfc9ee663fd9a1834698ac61fe9153462: Status 404 returned error can't find the container with id dbd095bd29437c58261c9f17a9f2c67bfc9ee663fd9a1834698ac61fe9153462 Nov 03 22:46:25.673872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:25.667863 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx"] Nov 03 22:46:25.734436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:25.734390 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-027a6fbe-xlftx"] Nov 03 22:46:26.067377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:26.055322 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" event={"ID":"01619c55-1af7-4068-8cbf-8ffd9da02445","Type":"ContainerStarted","Data":"b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170"} Nov 03 22:46:26.067377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:26.055398 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" event={"ID":"01619c55-1af7-4068-8cbf-8ffd9da02445","Type":"ContainerStarted","Data":"dbd095bd29437c58261c9f17a9f2c67bfc9ee663fd9a1834698ac61fe9153462"} Nov 03 22:46:26.067377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:26.056747 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:27.293360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:27.293303 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" podStartSLOduration=3.293289384 podStartE2EDuration="3.293289384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:26.172838484 +0000 UTC m=+1366.795146794" watchObservedRunningTime="2026-11-03 22:46:27.293289384 +0000 UTC m=+1367.915597692" Nov 03 22:46:27.299430 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:27.299396 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2"] Nov 03 22:46:27.421967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:27.421926 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj"] Nov 03 22:46:27.496759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:27.496715 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b791918-ca5a-4173-9c5b-f3e22681f22d" path="/var/lib/kubelet/pods/3b791918-ca5a-4173-9c5b-f3e22681f22d/volumes" Nov 03 22:46:32.051100 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.051061 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h"] Nov 03 22:46:32.097372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.097338 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc13563-bx77h"] Nov 03 22:46:32.300310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.300262 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" podUID="49d77b8c-8686-49b4-b35a-290dc6bafdfe" containerName="payload-processing" containerID="cri-o://24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba" gracePeriod=25 Nov 03 22:46:32.424300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.424250 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" podUID="c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" containerName="payload-pre-processing" containerID="cri-o://f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8" gracePeriod=25 Nov 03 22:46:32.603093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.603062 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:46:32.699959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.699920 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d77b8c-8686-49b4-b35a-290dc6bafdfe-kube-api-access-nhdnr\" (UniqueName: \"kubernetes.io/projected/49d77b8c-8686-49b4-b35a-290dc6bafdfe-kube-api-access-nhdnr\") pod \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " Nov 03 22:46:32.700189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.699999 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d77b8c-8686-49b4-b35a-290dc6bafdfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d77b8c-8686-49b4-b35a-290dc6bafdfe-plugins-config-volume\") pod \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\" (UID: \"49d77b8c-8686-49b4-b35a-290dc6bafdfe\") " Nov 03 22:46:32.701112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.701080 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d77b8c-8686-49b4-b35a-290dc6bafdfe-plugins-config-volume" pod "49d77b8c-8686-49b4-b35a-290dc6bafdfe" (UID: "49d77b8c-8686-49b4-b35a-290dc6bafdfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:32.710899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.710858 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d77b8c-8686-49b4-b35a-290dc6bafdfe-kube-api-access-nhdnr" pod "49d77b8c-8686-49b4-b35a-290dc6bafdfe" (UID: "49d77b8c-8686-49b4-b35a-290dc6bafdfe"). InnerVolumeSpecName "kube-api-access-nhdnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:32.753894 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.753860 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:46:32.802257 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.802220 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-plugins-config-volume\") pod \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " Nov 03 22:46:32.802444 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.802265 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-kube-api-access-7f9k7\" (UniqueName: \"kubernetes.io/projected/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-kube-api-access-7f9k7\") pod \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\" (UID: \"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e\") " Nov 03 22:46:32.802734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.802702 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-plugins-config-volume" pod "c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" (UID: "c8f9353f-3e8a-4ea9-8453-aa3baf7a524e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:32.802997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.802972 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:32.803057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.803002 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdnr\" (UniqueName: \"kubernetes.io/projected/49d77b8c-8686-49b4-b35a-290dc6bafdfe-kube-api-access-nhdnr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:32.803057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.803018 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d77b8c-8686-49b4-b35a-290dc6bafdfe-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:32.804469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.804447 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-kube-api-access-7f9k7" pod "c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" (UID: "c8f9353f-3e8a-4ea9-8453-aa3baf7a524e"). InnerVolumeSpecName "kube-api-access-7f9k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:32.906282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:32.906183 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9k7\" (UniqueName: \"kubernetes.io/projected/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e-kube-api-access-7f9k7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:33.153424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.153390 5272 generic.go:356] "Generic (PLEG): container finished" podID="c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" containerID="f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8" exitCode=0 Nov 03 22:46:33.153915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.153453 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" Nov 03 22:46:33.153915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.153471 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" event={"ID":"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e","Type":"ContainerDied","Data":"f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8"} Nov 03 22:46:33.153915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.153552 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj" event={"ID":"c8f9353f-3e8a-4ea9-8453-aa3baf7a524e","Type":"ContainerDied","Data":"0a6f883daf56273186342f9c380e8d35ffbe950fb97ec5130587a09613907206"} Nov 03 22:46:33.153915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.153578 5272 scope.go:122] "RemoveContainer" containerID="f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8" Nov 03 22:46:33.155116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.155090 5272 generic.go:356] "Generic (PLEG): container finished" podID="49d77b8c-8686-49b4-b35a-290dc6bafdfe" containerID="24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba" exitCode=0 Nov 03 22:46:33.155255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.155161 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" Nov 03 22:46:33.155255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.155179 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" event={"ID":"49d77b8c-8686-49b4-b35a-290dc6bafdfe","Type":"ContainerDied","Data":"24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba"} Nov 03 22:46:33.155255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.155235 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2" event={"ID":"49d77b8c-8686-49b4-b35a-290dc6bafdfe","Type":"ContainerDied","Data":"04845b7324ac863fe46cdf3c4170ac89f18f98f6287ade93f0dd9ea1ee1c343c"} Nov 03 22:46:33.167599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.167576 5272 scope.go:122] "RemoveContainer" containerID="f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8" Nov 03 22:46:33.167876 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:33.167856 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8\": container with ID starting with f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8 not found: ID does not exist" containerID="f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8" Nov 03 22:46:33.167957 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.167888 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8"} err="failed to get container status \"f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8\": rpc error: code = NotFound desc = could not find container \"f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8\": container with ID starting with f678670af4afb03468ba45bf1d45fbb973634ce09589aa34c967a702ce09f1d8 not found: ID does not exist" Nov 03 22:46:33.167957 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.167911 5272 scope.go:122] "RemoveContainer" containerID="24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba" Nov 03 22:46:33.177508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.177493 5272 scope.go:122] "RemoveContainer" containerID="24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba" Nov 03 22:46:33.177771 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:33.177753 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba\": container with ID starting with 24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba not found: ID does not exist" containerID="24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba" Nov 03 22:46:33.177844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.177779 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba"} err="failed to get container status \"24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba\": rpc error: code = NotFound desc = could not find container \"24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba\": container with ID starting with 24893517a7c3d04b3b6d929f87b7d45a28ba65405444f6bf40c7f6de6995bdba not found: ID does not exist" Nov 03 22:46:33.312696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.312662 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj"] Nov 03 22:46:33.358611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.358575 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6905007-5d7cf6b467-8hdjj"] Nov 03 22:46:33.419459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.419377 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2"] Nov 03 22:46:33.424026 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.423999 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6905007-65d4cdd474-s4nr2"] Nov 03 22:46:33.495364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.495332 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d77b8c-8686-49b4-b35a-290dc6bafdfe" path="/var/lib/kubelet/pods/49d77b8c-8686-49b4-b35a-290dc6bafdfe/volumes" Nov 03 22:46:33.495895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.495879 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f7dfee-3d60-4c21-9917-ee1f2e5ecce1" path="/var/lib/kubelet/pods/55f7dfee-3d60-4c21-9917-ee1f2e5ecce1/volumes" Nov 03 22:46:33.496344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:33.496331 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" path="/var/lib/kubelet/pods/c8f9353f-3e8a-4ea9-8453-aa3baf7a524e/volumes" Nov 03 22:46:34.135837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.129621 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd"] Nov 03 22:46:34.147525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.143781 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" containerName="payload-pre-processing" Nov 03 22:46:34.147525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.143831 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" containerName="payload-pre-processing" Nov 03 22:46:34.147525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.143869 5272 cpu_manager.go:414] "containerMap: removing container" podUID="49d77b8c-8686-49b4-b35a-290dc6bafdfe" containerName="payload-processing" Nov 03 22:46:34.147525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.143878 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d77b8c-8686-49b4-b35a-290dc6bafdfe" containerName="payload-processing" Nov 03 22:46:34.147525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.144118 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33bd25b-b162-47bd-aae3-e523f3ee31a7" containerName="istio-proxy" Nov 03 22:46:34.147525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.144135 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f9353f-3e8a-4ea9-8453-aa3baf7a524e" containerName="payload-pre-processing" Nov 03 22:46:34.147525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.144149 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d77b8c-8686-49b4-b35a-290dc6bafdfe" containerName="payload-processing" Nov 03 22:46:34.162141 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.162110 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.245892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.243329 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d18a5b05\"" Nov 03 22:46:34.245892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.243482 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d18a5b05-dockercfg-bz587\"" Nov 03 22:46:34.269970 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.251836 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3d556b-898c-4567-96c0-a4278f64f24a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.269970 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.251930 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pq8\" (UniqueName: \"kubernetes.io/projected/2a3d556b-898c-4567-96c0-a4278f64f24a-kube-api-access-g6pq8\") pod \"payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.334833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.334087 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd"] Nov 03 22:46:34.346885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.345620 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94"] Nov 03 22:46:34.346885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.356387 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:34.346885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.357083 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3d556b-898c-4567-96c0-a4278f64f24a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.346885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.357154 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pq8\" (UniqueName: \"kubernetes.io/projected/2a3d556b-898c-4567-96c0-a4278f64f24a-kube-api-access-g6pq8\") pod \"payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.346885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.358238 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3d556b-898c-4567-96c0-a4278f64f24a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.465590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.465176 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453aaffc-2a66-4e0a-8371-64755cd3c0d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:34.465590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.465376 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvvw\" (UniqueName: \"kubernetes.io/projected/453aaffc-2a66-4e0a-8371-64755cd3c0d5-kube-api-access-7dvvw\") pod \"payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:34.580829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.563355 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pq8\" (UniqueName: \"kubernetes.io/projected/2a3d556b-898c-4567-96c0-a4278f64f24a-kube-api-access-g6pq8\") pod \"payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.580829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.567826 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvvw\" (UniqueName: \"kubernetes.io/projected/453aaffc-2a66-4e0a-8371-64755cd3c0d5-kube-api-access-7dvvw\") pod \"payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:34.580829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.568053 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453aaffc-2a66-4e0a-8371-64755cd3c0d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:34.580829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.569441 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453aaffc-2a66-4e0a-8371-64755cd3c0d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:34.580829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.570149 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:34.654717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.654601 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvvw\" (UniqueName: \"kubernetes.io/projected/453aaffc-2a66-4e0a-8371-64755cd3c0d5-kube-api-access-7dvvw\") pod \"payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:34.657326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.655494 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94"] Nov 03 22:46:34.919499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.919456 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j"] Nov 03 22:46:34.937715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:34.937684 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:35.043679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.043619 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg"] Nov 03 22:46:35.053734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.053694 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd"] Nov 03 22:46:35.088768 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:46:35.088735 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3d556b_898c_4567_96c0_a4278f64f24a.slice/crio-b890d4d8694cad8cdb59664e0bfc2858e14f4d8d866adf94a71219cdc48cf92c WatchSource:0}: Error finding container b890d4d8694cad8cdb59664e0bfc2858e14f4d8d866adf94a71219cdc48cf92c: Status 404 returned error can't find the container with id b890d4d8694cad8cdb59664e0bfc2858e14f4d8d866adf94a71219cdc48cf92c Nov 03 22:46:35.321968 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.321932 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94"] Nov 03 22:46:35.326315 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:46:35.326277 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453aaffc_2a66_4e0a_8371_64755cd3c0d5.slice/crio-95460ee10576344f3a9f998195e6e8e35df46daa14127fffe2d4e69854cd8264 WatchSource:0}: Error finding container 95460ee10576344f3a9f998195e6e8e35df46daa14127fffe2d4e69854cd8264: Status 404 returned error can't find the container with id 95460ee10576344f3a9f998195e6e8e35df46daa14127fffe2d4e69854cd8264 Nov 03 22:46:35.657231 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.657181 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" event={"ID":"2a3d556b-898c-4567-96c0-a4278f64f24a","Type":"ContainerStarted","Data":"f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f"} Nov 03 22:46:35.657404 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.657246 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" event={"ID":"2a3d556b-898c-4567-96c0-a4278f64f24a","Type":"ContainerStarted","Data":"b890d4d8694cad8cdb59664e0bfc2858e14f4d8d866adf94a71219cdc48cf92c"} Nov 03 22:46:35.657404 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.657275 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:35.658721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.658695 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" event={"ID":"453aaffc-2a66-4e0a-8371-64755cd3c0d5","Type":"ContainerStarted","Data":"95460ee10576344f3a9f998195e6e8e35df46daa14127fffe2d4e69854cd8264"} Nov 03 22:46:35.767835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:35.767750 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" podStartSLOduration=1.767733424 podStartE2EDuration="1.767733424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:35.765000887 +0000 UTC m=+1376.387309196" watchObservedRunningTime="2026-11-03 22:46:35.767733424 +0000 UTC m=+1376.390041733" Nov 03 22:46:36.742259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:36.742201 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" event={"ID":"453aaffc-2a66-4e0a-8371-64755cd3c0d5","Type":"ContainerStarted","Data":"3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3"} Nov 03 22:46:36.935829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:36.930612 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" podStartSLOduration=2.930591527 podStartE2EDuration="2.930591527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:36.859264192 +0000 UTC m=+1377.481572501" watchObservedRunningTime="2026-11-03 22:46:36.930591527 +0000 UTC m=+1377.552899835" Nov 03 22:46:37.754480 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:37.754446 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:38.078704 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:38.078676 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:46:39.919819 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:39.919750 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" podUID="c8a23076-aa94-4308-9c33-a59ecb05b90a" containerName="payload-processing" containerID="cri-o://fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28" gracePeriod=25 Nov 03 22:46:40.044112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.044074 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" podUID="2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" containerName="payload-pre-processing" containerID="cri-o://d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a" gracePeriod=25 Nov 03 22:46:40.212828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.211551 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:46:40.266445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.266411 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a23076-aa94-4308-9c33-a59ecb05b90a-kube-api-access-9lhb4\" (UniqueName: \"kubernetes.io/projected/c8a23076-aa94-4308-9c33-a59ecb05b90a-kube-api-access-9lhb4\") pod \"c8a23076-aa94-4308-9c33-a59ecb05b90a\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " Nov 03 22:46:40.266634 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.266482 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a23076-aa94-4308-9c33-a59ecb05b90a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a23076-aa94-4308-9c33-a59ecb05b90a-plugins-config-volume\") pod \"c8a23076-aa94-4308-9c33-a59ecb05b90a\" (UID: \"c8a23076-aa94-4308-9c33-a59ecb05b90a\") " Nov 03 22:46:40.267242 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.267210 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a23076-aa94-4308-9c33-a59ecb05b90a-plugins-config-volume" pod "c8a23076-aa94-4308-9c33-a59ecb05b90a" (UID: "c8a23076-aa94-4308-9c33-a59ecb05b90a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:40.269253 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.269218 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a23076-aa94-4308-9c33-a59ecb05b90a-kube-api-access-9lhb4" pod "c8a23076-aa94-4308-9c33-a59ecb05b90a" (UID: "c8a23076-aa94-4308-9c33-a59ecb05b90a"). InnerVolumeSpecName "kube-api-access-9lhb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:40.310695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.310672 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:46:40.368282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.368251 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-kube-api-access-2p296\" (UniqueName: \"kubernetes.io/projected/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-kube-api-access-2p296\") pod \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " Nov 03 22:46:40.368449 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.368313 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-plugins-config-volume\") pod \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\" (UID: \"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965\") " Nov 03 22:46:40.368734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.368710 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-plugins-config-volume" pod "2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" (UID: "2cb9bf3c-5a18-4cb5-8fca-9f4baa200965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:40.368904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.368888 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lhb4\" (UniqueName: \"kubernetes.io/projected/c8a23076-aa94-4308-9c33-a59ecb05b90a-kube-api-access-9lhb4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:40.368969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.368911 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a23076-aa94-4308-9c33-a59ecb05b90a-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:40.368969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.368925 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:40.370632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.370602 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-kube-api-access-2p296" pod "2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" (UID: "2cb9bf3c-5a18-4cb5-8fca-9f4baa200965"). InnerVolumeSpecName "kube-api-access-2p296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:40.470540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.470445 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p296\" (UniqueName: \"kubernetes.io/projected/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965-kube-api-access-2p296\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:46:40.781277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.781186 5272 generic.go:356] "Generic (PLEG): container finished" podID="c8a23076-aa94-4308-9c33-a59ecb05b90a" containerID="fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28" exitCode=0 Nov 03 22:46:40.781277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.781224 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" Nov 03 22:46:40.781509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.781271 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" event={"ID":"c8a23076-aa94-4308-9c33-a59ecb05b90a","Type":"ContainerDied","Data":"fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28"} Nov 03 22:46:40.781509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.781332 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j" event={"ID":"c8a23076-aa94-4308-9c33-a59ecb05b90a","Type":"ContainerDied","Data":"be824cad058ba686e3bbd995088ec404c7c6fee932d8b35a2994ea9f25079351"} Nov 03 22:46:40.781509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.781362 5272 scope.go:122] "RemoveContainer" containerID="fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28" Nov 03 22:46:40.782978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.782954 5272 generic.go:356] "Generic (PLEG): container finished" podID="2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" containerID="d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a" exitCode=0 Nov 03 22:46:40.783092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.783002 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" Nov 03 22:46:40.783172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.783135 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" event={"ID":"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965","Type":"ContainerDied","Data":"d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a"} Nov 03 22:46:40.783240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.783179 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg" event={"ID":"2cb9bf3c-5a18-4cb5-8fca-9f4baa200965","Type":"ContainerDied","Data":"943bb0d90701270c95cde4e7678c8c0cb79bcff33cad859b3f031fba6f1197e4"} Nov 03 22:46:40.793881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.793864 5272 scope.go:122] "RemoveContainer" containerID="fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28" Nov 03 22:46:40.794153 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:40.794130 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28\": container with ID starting with fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28 not found: ID does not exist" containerID="fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28" Nov 03 22:46:40.794214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.794161 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28"} err="failed to get container status \"fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28\": rpc error: code = NotFound desc = could not find container \"fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28\": container with ID starting with fb7f1a31ca5649995fe7a5823cccd123c51de700c129edf1e4b1067d24f70d28 not found: ID does not exist" Nov 03 22:46:40.794214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.794178 5272 scope.go:122] "RemoveContainer" containerID="d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a" Nov 03 22:46:40.804499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.804477 5272 scope.go:122] "RemoveContainer" containerID="d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a" Nov 03 22:46:40.804758 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:40.804737 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a\": container with ID starting with d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a not found: ID does not exist" containerID="d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a" Nov 03 22:46:40.804857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.804769 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a"} err="failed to get container status \"d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a\": rpc error: code = NotFound desc = could not find container \"d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a\": container with ID starting with d6f0ef2104ccff9cf3581432a751551a3363e04c96e711f6a6bcdb7808ff585a not found: ID does not exist" Nov 03 22:46:40.951831 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.947450 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg"] Nov 03 22:46:40.956824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:40.956192 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c082ab23-5df95f8c-dvxqg"] Nov 03 22:46:41.059798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:41.059759 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j"] Nov 03 22:46:41.064417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:41.064394 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c082ab23-876b49bfb-6f94j"] Nov 03 22:46:41.479217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:41.479133 5272 scope.go:122] "RemoveContainer" containerID="27a268226c3ce438901df4a340a74d9acdb7e2a302fd21afce0c30be5d4bd2f9" Nov 03 22:46:41.496385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:41.496354 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" path="/var/lib/kubelet/pods/2cb9bf3c-5a18-4cb5-8fca-9f4baa200965/volumes" Nov 03 22:46:41.496884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:41.496868 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a23076-aa94-4308-9c33-a59ecb05b90a" path="/var/lib/kubelet/pods/c8a23076-aa94-4308-9c33-a59ecb05b90a/volumes" Nov 03 22:46:46.747658 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:46.747621 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:46:48.766611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:48.766579 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:46:54.502409 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:46:54.502372 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:46:58.049316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:58.049279 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj"] Nov 03 22:46:58.095531 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:58.095486 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d52e2b4a-x8pfj"] Nov 03 22:46:59.494648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:46:59.494610 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349bc540-633a-4a02-ad94-54d25cdbf610" path="/var/lib/kubelet/pods/349bc540-633a-4a02-ad94-54d25cdbf610/volumes" Nov 03 22:47:01.999897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:01.999858 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv"] Nov 03 22:47:02.021977 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:02.021940 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c082ab23-k2qjv"] Nov 03 22:47:03.047522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.047487 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj"] Nov 03 22:47:03.101434 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.101398 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113434f6-mxsrj"] Nov 03 22:47:03.293118 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.293080 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6"] Nov 03 22:47:03.297497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.297459 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6905007-5gtd6"] Nov 03 22:47:03.480833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.480733 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2"] Nov 03 22:47:03.481084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.481043 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" containerID="cri-o://990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51" gracePeriod=30 Nov 03 22:47:03.498723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.498690 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d" path="/var/lib/kubelet/pods/0bdafdd1-8b0f-4ad1-9985-da859d5f3e5d/volumes" Nov 03 22:47:03.499488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.499465 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26337f4e-2fd4-405d-a69d-dabbd6824483" path="/var/lib/kubelet/pods/26337f4e-2fd4-405d-a69d-dabbd6824483/volumes" Nov 03 22:47:03.500177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:03.500155 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57204af1-6a8c-4741-bb82-9e6d178378d4" path="/var/lib/kubelet/pods/57204af1-6a8c-4741-bb82-9e6d178378d4/volumes" Nov 03 22:47:05.821507 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.821472 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc"] Nov 03 22:47:05.822222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.822205 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c8a23076-aa94-4308-9c33-a59ecb05b90a" containerName="payload-processing" Nov 03 22:47:05.822222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.822222 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a23076-aa94-4308-9c33-a59ecb05b90a" containerName="payload-processing" Nov 03 22:47:05.822324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.822245 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" containerName="payload-pre-processing" Nov 03 22:47:05.822324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.822251 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" containerName="payload-pre-processing" Nov 03 22:47:05.822400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.822375 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a23076-aa94-4308-9c33-a59ecb05b90a" containerName="payload-processing" Nov 03 22:47:05.822400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.822385 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb9bf3c-5a18-4cb5-8fca-9f4baa200965" containerName="payload-pre-processing" Nov 03 22:47:05.828087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.828060 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.878191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.878164 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-12450a06-openshift-default-dockercfg-wvwz8\"" Nov 03 22:47:05.932243 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.932207 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc"] Nov 03 22:47:05.940085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940045 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940102 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940132 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940208 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940264 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940288 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32b4f71-9f49-4f92-b766-88ce76449123-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940320 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32b4f71-9f49-4f92-b766-88ce76449123-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940444 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940354 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:05.940444 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:05.940397 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p777m\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-kube-api-access-p777m\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.043131 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.043081 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.044673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.043822 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p777m\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-kube-api-access-p777m\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.045470 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.045443 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.045992 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.045973 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.046410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.046391 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.046972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.046954 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.047474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.047457 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.048018 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.048000 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32b4f71-9f49-4f92-b766-88ce76449123-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.048833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.048814 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32b4f71-9f49-4f92-b766-88ce76449123-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.049377 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.045893 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.049494 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.047798 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.049579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.046774 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.049650 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.048720 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32b4f71-9f49-4f92-b766-88ce76449123-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.049717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.043702 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.054228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.054207 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.098897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.086326 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32b4f71-9f49-4f92-b766-88ce76449123-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.155826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.152261 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p777m\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-kube-api-access-p777m\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.196827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.189873 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.442746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.442641 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:06.883045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.882840 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc"] Nov 03 22:47:06.886291 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:06.886265 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32b4f71_9f49_4f92_b766_88ce76449123.slice/crio-53731fff8c6664aba2d5e39082dffa432b7b4e8025b722bf674f2e50142d7282 WatchSource:0}: Error finding container 53731fff8c6664aba2d5e39082dffa432b7b4e8025b722bf674f2e50142d7282: Status 404 returned error can't find the container with id 53731fff8c6664aba2d5e39082dffa432b7b4e8025b722bf674f2e50142d7282 Nov 03 22:47:06.889033 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.889003 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:06.889114 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.889068 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:06.889153 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:06.889120 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:07.245561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:07.245456 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" event={"ID":"f32b4f71-9f49-4f92-b766-88ce76449123","Type":"ContainerStarted","Data":"5aab74f971329a16fb7ee278f4f3db2bfbea96d0eddcb514313eedc9eab7986d"} Nov 03 22:47:07.245561 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:07.245524 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" event={"ID":"f32b4f71-9f49-4f92-b766-88ce76449123","Type":"ContainerStarted","Data":"53731fff8c6664aba2d5e39082dffa432b7b4e8025b722bf674f2e50142d7282"} Nov 03 22:47:07.359912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:07.359793 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podStartSLOduration=2.359779899 podStartE2EDuration="2.359779899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:07.358535231 +0000 UTC m=+1407.980843540" watchObservedRunningTime="2026-11-03 22:47:07.359779899 +0000 UTC m=+1407.982088206" Nov 03 22:47:07.443268 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:07.443234 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:07.449577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:07.449554 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:08.293822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.289419 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:08.323827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.320346 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:47:08.819556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.819529 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:47:08.888271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888189 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-data\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.888446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888299 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-certs\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.888446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888367 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-envoy\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.888446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888440 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-kube-api-access-qqb2r\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-kube-api-access-qqb2r\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888501 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-data" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888572 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-credential-socket\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888640 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-certs" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888649 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f08872-278e-4f08-9a4d-11e1e223321a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f08872-278e-4f08-9a4d-11e1e223321a-istiod-ca-cert\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888769 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-socket\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.888923 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-istio-token\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-istio-token\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889014 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77f08872-278e-4f08-9a4d-11e1e223321a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f08872-278e-4f08-9a4d-11e1e223321a-istio-podinfo\") pod \"77f08872-278e-4f08-9a4d-11e1e223321a\" (UID: \"77f08872-278e-4f08-9a4d-11e1e223321a\") " Nov 03 22:47:08.889304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889096 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f08872-278e-4f08-9a4d-11e1e223321a-istiod-ca-cert" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:08.889849 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889340 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-socket" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:08.889849 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889472 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-credential-socket" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:08.889984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889903 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.889984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889925 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f08872-278e-4f08-9a4d-11e1e223321a-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.889984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889937 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.889984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889949 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.889984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.889962 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.891947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.891914 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-istio-token" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:08.896865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.896824 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77f08872-278e-4f08-9a4d-11e1e223321a-istio-podinfo" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:08.897003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.896980 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-kube-api-access-qqb2r" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "kube-api-access-qqb2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:08.898205 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.898177 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-envoy" pod "77f08872-278e-4f08-9a4d-11e1e223321a" (UID: "77f08872-278e-4f08-9a4d-11e1e223321a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:08.991750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.991705 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f08872-278e-4f08-9a4d-11e1e223321a-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.991750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.991740 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqb2r\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-kube-api-access-qqb2r\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.991750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.991752 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f08872-278e-4f08-9a4d-11e1e223321a-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:08.991750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:08.991761 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f08872-278e-4f08-9a4d-11e1e223321a-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:09.298252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.298219 5272 generic.go:356] "Generic (PLEG): container finished" podID="77f08872-278e-4f08-9a4d-11e1e223321a" containerID="990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51" exitCode=0 Nov 03 22:47:09.298769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.298274 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" Nov 03 22:47:09.298769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.298306 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" event={"ID":"77f08872-278e-4f08-9a4d-11e1e223321a","Type":"ContainerDied","Data":"990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51"} Nov 03 22:47:09.298769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.298365 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2" event={"ID":"77f08872-278e-4f08-9a4d-11e1e223321a","Type":"ContainerDied","Data":"6d8f4d575b0cbbbe9f800d7915f06881beaa00389812baad9ef6ab7aeea1970a"} Nov 03 22:47:09.298769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.298388 5272 scope.go:122] "RemoveContainer" containerID="990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51" Nov 03 22:47:09.310486 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.310471 5272 scope.go:122] "RemoveContainer" containerID="7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60" Nov 03 22:47:09.321304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.321283 5272 scope.go:122] "RemoveContainer" containerID="990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51" Nov 03 22:47:09.321577 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:47:09.321558 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51\": container with ID starting with 990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51 not found: ID does not exist" containerID="990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51" Nov 03 22:47:09.321627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.321587 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51"} err="failed to get container status \"990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51\": rpc error: code = NotFound desc = could not find container \"990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51\": container with ID starting with 990308cb53b751e456658c7656048d3f7a5210ca23d34748750e79685401ed51 not found: ID does not exist" Nov 03 22:47:09.321627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.321607 5272 scope.go:122] "RemoveContainer" containerID="7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60" Nov 03 22:47:09.321866 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:47:09.321848 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60\": container with ID starting with 7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60 not found: ID does not exist" containerID="7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60" Nov 03 22:47:09.321916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.321873 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60"} err="failed to get container status \"7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60\": rpc error: code = NotFound desc = could not find container \"7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60\": container with ID starting with 7ef6cc4f7d2ee371c0f826d4c4dddd526412da965abcf61c639790ba53d22b60 not found: ID does not exist" Nov 03 22:47:09.460716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.460680 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2"] Nov 03 22:47:09.506305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:09.506274 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c082ab23-openshift-default-7f9b898b47-fpps2"] Nov 03 22:47:11.555223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:11.552703 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" path="/var/lib/kubelet/pods/77f08872-278e-4f08-9a4d-11e1e223321a/volumes" Nov 03 22:47:13.876312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.876271 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k"] Nov 03 22:47:13.877106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.877086 5272 cpu_manager.go:414] "containerMap: removing container" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" Nov 03 22:47:13.877106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.877106 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" Nov 03 22:47:13.877292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.877133 5272 cpu_manager.go:414] "containerMap: removing container" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" Nov 03 22:47:13.877292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.877138 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" Nov 03 22:47:13.877292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.877270 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" Nov 03 22:47:13.882795 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.882777 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:13.913249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.913221 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641a4a4c-cce4-49e8-bedd-1977115c8325-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:13.913450 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.913427 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcsm\" (UniqueName: \"kubernetes.io/projected/641a4a4c-cce4-49e8-bedd-1977115c8325-kube-api-access-tvcsm\") pod \"payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:13.937383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.937356 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k"] Nov 03 22:47:13.977898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.977861 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-12450a06\"" Nov 03 22:47:13.979110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:13.979016 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-12450a06-dockercfg-6zscm\"" Nov 03 22:47:14.024914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.023796 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcsm\" (UniqueName: \"kubernetes.io/projected/641a4a4c-cce4-49e8-bedd-1977115c8325-kube-api-access-tvcsm\") pod \"payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:14.047900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.024160 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641a4a4c-cce4-49e8-bedd-1977115c8325-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:14.047900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.040016 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641a4a4c-cce4-49e8-bedd-1977115c8325-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:14.132893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.131570 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcsm\" (UniqueName: \"kubernetes.io/projected/641a4a4c-cce4-49e8-bedd-1977115c8325-kube-api-access-tvcsm\") pod \"payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:14.247306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.247020 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:14.333871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.290933 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk"] Nov 03 22:47:14.333871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.311502 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f08872-278e-4f08-9a4d-11e1e223321a" containerName="istio-proxy" Nov 03 22:47:14.333871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.319224 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.373751 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.372894 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbf7\" (UniqueName: \"kubernetes.io/projected/0ff84017-6a02-412b-aa89-8e1bc502fc84-kube-api-access-wdbf7\") pod \"payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.373751 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.373084 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff84017-6a02-412b-aa89-8e1bc502fc84-plugins-config-volume\") pod \"payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.381723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.381683 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk"] Nov 03 22:47:14.475062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.475010 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff84017-6a02-412b-aa89-8e1bc502fc84-plugins-config-volume\") pod \"payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.475252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.475229 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbf7\" (UniqueName: \"kubernetes.io/projected/0ff84017-6a02-412b-aa89-8e1bc502fc84-kube-api-access-wdbf7\") pod \"payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.476027 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.475999 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff84017-6a02-412b-aa89-8e1bc502fc84-plugins-config-volume\") pod \"payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.533505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.533472 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbf7\" (UniqueName: \"kubernetes.io/projected/0ff84017-6a02-412b-aa89-8e1bc502fc84-kube-api-access-wdbf7\") pod \"payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.701094 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.701010 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:14.750577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:14.750411 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k"] Nov 03 22:47:14.754212 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:14.754183 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641a4a4c_cce4_49e8_bedd_1977115c8325.slice/crio-188ced8d4e526b741e0b4ddc0b4c3aae9b9b6c291ef7526fdb9b615e56841406 WatchSource:0}: Error finding container 188ced8d4e526b741e0b4ddc0b4c3aae9b9b6c291ef7526fdb9b615e56841406: Status 404 returned error can't find the container with id 188ced8d4e526b741e0b4ddc0b4c3aae9b9b6c291ef7526fdb9b615e56841406 Nov 03 22:47:15.092681 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.092642 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk"] Nov 03 22:47:15.099187 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:15.099154 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff84017_6a02_412b_aa89_8e1bc502fc84.slice/crio-c20efd31796270b9543ef2f8fcef6a8926d72e2b14d6af2e7745733f39d57166 WatchSource:0}: Error finding container c20efd31796270b9543ef2f8fcef6a8926d72e2b14d6af2e7745733f39d57166: Status 404 returned error can't find the container with id c20efd31796270b9543ef2f8fcef6a8926d72e2b14d6af2e7745733f39d57166 Nov 03 22:47:15.386490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.386405 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" event={"ID":"0ff84017-6a02-412b-aa89-8e1bc502fc84","Type":"ContainerStarted","Data":"0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105"} Nov 03 22:47:15.386490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.386457 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" event={"ID":"0ff84017-6a02-412b-aa89-8e1bc502fc84","Type":"ContainerStarted","Data":"c20efd31796270b9543ef2f8fcef6a8926d72e2b14d6af2e7745733f39d57166"} Nov 03 22:47:15.386735 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.386519 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:15.388433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.388408 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" event={"ID":"641a4a4c-cce4-49e8-bedd-1977115c8325","Type":"ContainerStarted","Data":"2ebbf6c29beaa6275254068335e482a2cfeb0fcda857bfc130fa636943b50db3"} Nov 03 22:47:15.388585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.388445 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" event={"ID":"641a4a4c-cce4-49e8-bedd-1977115c8325","Type":"ContainerStarted","Data":"188ced8d4e526b741e0b4ddc0b4c3aae9b9b6c291ef7526fdb9b615e56841406"} Nov 03 22:47:15.388585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.388490 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:15.495954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.495906 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" podStartSLOduration=2.49589178 podStartE2EDuration="2.49589178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:15.494473952 +0000 UTC m=+1416.116782260" watchObservedRunningTime="2026-11-03 22:47:15.49589178 +0000 UTC m=+1416.118200088" Nov 03 22:47:15.630830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:15.624225 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" podStartSLOduration=2.62420048 podStartE2EDuration="2.62420048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:15.603479565 +0000 UTC m=+1416.225787877" watchObservedRunningTime="2026-11-03 22:47:15.62420048 +0000 UTC m=+1416.246508789" Nov 03 22:47:16.989189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:16.989147 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:47:18.051158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:18.051124 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn"] Nov 03 22:47:18.097595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:18.097560 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3889e75f-qlhzn"] Nov 03 22:47:19.497235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:19.497202 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ada9ed-67f8-44da-917e-4363da6de8d0" path="/var/lib/kubelet/pods/f0ada9ed-67f8-44da-917e-4363da6de8d0/volumes" Nov 03 22:47:23.256233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.256200 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc"] Nov 03 22:47:23.262314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.262285 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.352488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.352442 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvbwm\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-kube-api-access-bvbwm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.352678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.352519 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.352678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.352592 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.352832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.352744 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.352832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.352796 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.353010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.352991 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.353070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.353024 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e991-5375-41b5-90ca-35b6f7bb2678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.353130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.353099 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.353188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.353131 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.433240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.433205 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj"] Nov 03 22:47:23.438200 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.438170 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.454177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454141 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.454411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454220 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.454411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454317 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-certs\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.454411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454400 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-token\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.454597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454456 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjpc\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-kube-api-access-sxjpc\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.454597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454520 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.454597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454549 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e991-5375-41b5-90ca-35b6f7bb2678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.454769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454605 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-envoy\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.454769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454657 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.454769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454697 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.454769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454727 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-socket\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.454769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454764 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvbwm\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-kube-api-access-bvbwm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.455112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454826 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-credential-socket\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.455112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454857 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-data\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.455112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.454918 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.455112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.455008 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.455112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.455085 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-podinfo\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.455371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.455114 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed69d8c-da98-4c76-8d68-b475ebb41024-istiod-ca-cert\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.455578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.455551 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.455877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.455850 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.456935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.456148 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.456935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.456745 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.458604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.457921 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e991-5375-41b5-90ca-35b6f7bb2678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.461051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.461023 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.482203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.481852 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.498712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.498678 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc"] Nov 03 22:47:23.498867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.498730 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj"] Nov 03 22:47:23.515595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.515538 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvbwm\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-kube-api-access-bvbwm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.515723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.515659 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-vhkwc\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557067 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-token\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557151 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjpc\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-kube-api-access-sxjpc\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557246 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-envoy\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557307 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-socket\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557375 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-credential-socket\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557408 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-data\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557538 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-podinfo\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557564 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed69d8c-da98-4c76-8d68-b475ebb41024-istiod-ca-cert\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.557738 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-certs\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.558365 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-certs\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.568595 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-data\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.569023 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-socket\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.569275 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-credential-socket\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.579316 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed69d8c-da98-4c76-8d68-b475ebb41024-istiod-ca-cert\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.581826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.580660 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:23.583255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.583127 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a03fee-openshift-default-dockercfg-rfn8q\"" Nov 03 22:47:23.642828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.638339 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-envoy\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.693840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.682308 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-token\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.693840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.684741 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjpc\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-kube-api-access-sxjpc\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.716865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.713386 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-podinfo\") pod \"e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:23.758824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:23.758148 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:24.125366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.125323 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv"] Nov 03 22:47:24.130202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.130161 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566a5589-k85kv"] Nov 03 22:47:24.317762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.317733 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc"] Nov 03 22:47:24.321507 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:24.321482 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b12e991_5375_41b5_90ca_35b6f7bb2678.slice/crio-2601afa66d90b1997bfee1981bf0e8943cdffcbd567acc8b3824eef2e9515975 WatchSource:0}: Error finding container 2601afa66d90b1997bfee1981bf0e8943cdffcbd567acc8b3824eef2e9515975: Status 404 returned error can't find the container with id 2601afa66d90b1997bfee1981bf0e8943cdffcbd567acc8b3824eef2e9515975 Nov 03 22:47:24.324040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.324003 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.324149 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.324086 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.324149 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.324134 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.436125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.436093 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj"] Nov 03 22:47:24.440135 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:24.440102 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed69d8c_da98_4c76_8d68_b475ebb41024.slice/crio-91d13ca83274132e5fdce7ba64dc860331eb0b38c77b10bad874dd44cdcf3505 WatchSource:0}: Error finding container 91d13ca83274132e5fdce7ba64dc860331eb0b38c77b10bad874dd44cdcf3505: Status 404 returned error can't find the container with id 91d13ca83274132e5fdce7ba64dc860331eb0b38c77b10bad874dd44cdcf3505 Nov 03 22:47:24.443476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.443445 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.443575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.443510 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.443575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.443537 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:47:24.504972 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:47:24.504933 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:47:24.563025 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.562993 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" event={"ID":"1ed69d8c-da98-4c76-8d68-b475ebb41024","Type":"ContainerStarted","Data":"3fe0c16cc46fb6b435a46726669df7003222fc6b2f432d77b83a33f2d4078501"} Nov 03 22:47:24.563177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.563053 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" event={"ID":"1ed69d8c-da98-4c76-8d68-b475ebb41024","Type":"ContainerStarted","Data":"91d13ca83274132e5fdce7ba64dc860331eb0b38c77b10bad874dd44cdcf3505"} Nov 03 22:47:24.565947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.565914 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" event={"ID":"8b12e991-5375-41b5-90ca-35b6f7bb2678","Type":"ContainerStarted","Data":"98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d"} Nov 03 22:47:24.566106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.565972 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" event={"ID":"8b12e991-5375-41b5-90ca-35b6f7bb2678","Type":"ContainerStarted","Data":"2601afa66d90b1997bfee1981bf0e8943cdffcbd567acc8b3824eef2e9515975"} Nov 03 22:47:24.583961 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.583920 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:24.713879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.710699 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podStartSLOduration=1.710680302 podStartE2EDuration="1.710680302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:24.708095522 +0000 UTC m=+1425.330403830" watchObservedRunningTime="2026-11-03 22:47:24.710680302 +0000 UTC m=+1425.332988613" Nov 03 22:47:24.760939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.759273 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:24.852826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:24.850504 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podStartSLOduration=1.850482123 podStartE2EDuration="1.850482123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:24.825627226 +0000 UTC m=+1425.447935552" watchObservedRunningTime="2026-11-03 22:47:24.850482123 +0000 UTC m=+1425.472790441" Nov 03 22:47:25.495708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:25.495671 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a" path="/var/lib/kubelet/pods/3c0f39f4-0c5f-4b5a-8dd6-51a84c617a9a/volumes" Nov 03 22:47:25.585446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:25.585415 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:25.585603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:25.585465 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:25.773814 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:25.773724 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:26.428797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:26.428768 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:47:26.437710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:26.437690 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:47:26.650827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:26.637563 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:26.650827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:26.637647 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:26.910829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:26.909620 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:26.911873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:26.911153 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:47:27.610060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:27.609709 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:27.610060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:27.609784 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:28.200655 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.200619 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d"] Nov 03 22:47:28.205298 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.205271 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.274864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.274141 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:47:28.320843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.320815 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:47:28.320997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.320854 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:47:28.320997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.320860 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:47:28.325771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.325748 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d"] Nov 03 22:47:28.338201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.338175 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.338314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.338242 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.338512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.338490 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dvc\" (UniqueName: \"kubernetes.io/projected/05346c7e-1f40-4a9a-a077-f2bfcd29eece-kube-api-access-b4dvc\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.440406 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.440372 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.440568 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.440432 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.440617 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.440568 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dvc\" (UniqueName: \"kubernetes.io/projected/05346c7e-1f40-4a9a-a077-f2bfcd29eece-kube-api-access-b4dvc\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.441183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.441155 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.442695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.442666 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.497852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.497778 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dvc\" (UniqueName: \"kubernetes.io/projected/05346c7e-1f40-4a9a-a077-f2bfcd29eece-kube-api-access-b4dvc\") pod \"maas-api-revoke-sub-e54a46a2-jqc4d\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.522065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.522036 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:28.583552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.583517 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:28.583706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.583587 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:28.937131 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:28.937094 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d"] Nov 03 22:47:28.940973 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:28.940941 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05346c7e_1f40_4a9a_a077_f2bfcd29eece.slice/crio-ddd2a921776c5c1101118ad9e2134dc571682edd37df47c678c7e73eac404c9b WatchSource:0}: Error finding container ddd2a921776c5c1101118ad9e2134dc571682edd37df47c678c7e73eac404c9b: Status 404 returned error can't find the container with id ddd2a921776c5c1101118ad9e2134dc571682edd37df47c678c7e73eac404c9b Nov 03 22:47:29.583931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:29.583896 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:29.584349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:29.583949 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:29.935670 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:29.935636 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" event={"ID":"05346c7e-1f40-4a9a-a077-f2bfcd29eece","Type":"ContainerStarted","Data":"595f09f5175feb646db56beeae68280cad3a905488ffa2a959d2d688f214e029"} Nov 03 22:47:29.935898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:29.935693 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" event={"ID":"05346c7e-1f40-4a9a-a077-f2bfcd29eece","Type":"ContainerStarted","Data":"ddd2a921776c5c1101118ad9e2134dc571682edd37df47c678c7e73eac404c9b"} Nov 03 22:47:30.068864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.063524 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" podStartSLOduration=2.063490962 podStartE2EDuration="2.063490962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:30.063029601 +0000 UTC m=+1430.685337908" watchObservedRunningTime="2026-11-03 22:47:30.063490962 +0000 UTC m=+1430.685799271" Nov 03 22:47:30.590888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.589960 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:30.590888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.590025 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:30.768967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.768930 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2"] Nov 03 22:47:30.775116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.775087 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.804120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.804081 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.804255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.804166 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492c5\" (UniqueName: \"kubernetes.io/projected/26c3efad-74fe-49aa-8544-6d651cd87953-kube-api-access-492c5\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.804255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.804194 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.806311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.806286 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2"] Nov 03 22:47:30.905496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.905409 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.905496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.905460 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492c5\" (UniqueName: \"kubernetes.io/projected/26c3efad-74fe-49aa-8544-6d651cd87953-kube-api-access-492c5\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.905756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.905569 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.906322 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.906298 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.907877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.907857 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.970436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.970401 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-492c5\" (UniqueName: \"kubernetes.io/projected/26c3efad-74fe-49aa-8544-6d651cd87953-kube-api-access-492c5\") pod \"maas-api-revoke-sub-c7e88602-wpcf2\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:30.973099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.973073 5272 generic.go:356] "Generic (PLEG): container finished" podID="05346c7e-1f40-4a9a-a077-f2bfcd29eece" containerID="595f09f5175feb646db56beeae68280cad3a905488ffa2a959d2d688f214e029" exitCode=0 Nov 03 22:47:30.973238 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:30.973149 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" event={"ID":"05346c7e-1f40-4a9a-a077-f2bfcd29eece","Type":"ContainerDied","Data":"595f09f5175feb646db56beeae68280cad3a905488ffa2a959d2d688f214e029"} Nov 03 22:47:31.090161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:31.090129 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:31.697829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:31.671949 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:31.697829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:31.672026 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:31.812827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:31.806118 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz"] Nov 03 22:47:31.846420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:31.840889 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:31.924031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:31.897630 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a03fee\"" Nov 03 22:47:31.924031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:31.897914 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a03fee-dockercfg-gn7xv\"" Nov 03 22:47:32.049822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.047720 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d94103-8548-4474-93d1-bc5a003d53b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:32.049822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.047841 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmz56\" (UniqueName: \"kubernetes.io/projected/72d94103-8548-4474-93d1-bc5a003d53b4-kube-api-access-bmz56\") pod \"payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:32.100145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.094787 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz"] Nov 03 22:47:32.101072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.100976 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2"] Nov 03 22:47:32.106436 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:32.106373 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c3efad_74fe_49aa_8544_6d651cd87953.slice/crio-4d792e58197dadeeabb43a57d68ae34907396892e5b4c0d4db0ffc5596283eb6 WatchSource:0}: Error finding container 4d792e58197dadeeabb43a57d68ae34907396892e5b4c0d4db0ffc5596283eb6: Status 404 returned error can't find the container with id 4d792e58197dadeeabb43a57d68ae34907396892e5b4c0d4db0ffc5596283eb6 Nov 03 22:47:32.107476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.107205 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk"] Nov 03 22:47:32.121709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.121594 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk"] Nov 03 22:47:32.122282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.122182 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.148971 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.148941 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d94103-8548-4474-93d1-bc5a003d53b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:32.149115 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.148995 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmz56\" (UniqueName: \"kubernetes.io/projected/72d94103-8548-4474-93d1-bc5a003d53b4-kube-api-access-bmz56\") pod \"payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:32.149601 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.149578 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d94103-8548-4474-93d1-bc5a003d53b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:32.207585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.207524 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmz56\" (UniqueName: \"kubernetes.io/projected/72d94103-8548-4474-93d1-bc5a003d53b4-kube-api-access-bmz56\") pod \"payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:32.250514 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.250399 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8rn\" (UniqueName: \"kubernetes.io/projected/dac539b8-4b24-4c8e-ab9c-aabaa943b303-kube-api-access-tz8rn\") pod \"payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.250681 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.250656 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac539b8-4b24-4c8e-ab9c-aabaa943b303-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.296201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.296166 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:32.352293 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.352257 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-xks-ca\") pod \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " Nov 03 22:47:32.352455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.352327 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-openshift-ca\") pod \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " Nov 03 22:47:32.352455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.352393 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05346c7e-1f40-4a9a-a077-f2bfcd29eece-kube-api-access-b4dvc\" (UniqueName: \"kubernetes.io/projected/05346c7e-1f40-4a9a-a077-f2bfcd29eece-kube-api-access-b4dvc\") pod \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\" (UID: \"05346c7e-1f40-4a9a-a077-f2bfcd29eece\") " Nov 03 22:47:32.352659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.352637 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-openshift-ca" pod "05346c7e-1f40-4a9a-a077-f2bfcd29eece" (UID: "05346c7e-1f40-4a9a-a077-f2bfcd29eece"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:32.352762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.352741 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8rn\" (UniqueName: \"kubernetes.io/projected/dac539b8-4b24-4c8e-ab9c-aabaa943b303-kube-api-access-tz8rn\") pod \"payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.352930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.352911 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac539b8-4b24-4c8e-ab9c-aabaa943b303-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.353121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.353105 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:32.353834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.353782 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac539b8-4b24-4c8e-ab9c-aabaa943b303-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.354369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.354347 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-xks-ca" pod "05346c7e-1f40-4a9a-a077-f2bfcd29eece" (UID: "05346c7e-1f40-4a9a-a077-f2bfcd29eece"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:32.354484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.354461 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05346c7e-1f40-4a9a-a077-f2bfcd29eece-kube-api-access-b4dvc" pod "05346c7e-1f40-4a9a-a077-f2bfcd29eece" (UID: "05346c7e-1f40-4a9a-a077-f2bfcd29eece"). InnerVolumeSpecName "kube-api-access-b4dvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:32.393912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.393875 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:32.414973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.414948 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8rn\" (UniqueName: \"kubernetes.io/projected/dac539b8-4b24-4c8e-ab9c-aabaa943b303-kube-api-access-tz8rn\") pod \"payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.454667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.454633 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05346c7e-1f40-4a9a-a077-f2bfcd29eece-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:32.454667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.454671 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4dvc\" (UniqueName: \"kubernetes.io/projected/05346c7e-1f40-4a9a-a077-f2bfcd29eece-kube-api-access-b4dvc\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:32.468519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.468400 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:32.583592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.583561 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:32.583737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.583622 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:32.791843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.791797 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz"] Nov 03 22:47:32.794553 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:32.794528 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d94103_8548_4474_93d1_bc5a003d53b4.slice/crio-646128cfd64a9eb450fb2552e8b63bcce1138182ee3acb75649f35f1545eb998 WatchSource:0}: Error finding container 646128cfd64a9eb450fb2552e8b63bcce1138182ee3acb75649f35f1545eb998: Status 404 returned error can't find the container with id 646128cfd64a9eb450fb2552e8b63bcce1138182ee3acb75649f35f1545eb998 Nov 03 22:47:32.894438 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:32.894409 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk"] Nov 03 22:47:32.898503 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:47:32.898475 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac539b8_4b24_4c8e_ab9c_aabaa943b303.slice/crio-902419a68fdfbc5c3f6942c930833b68e061f24ecbf474d4aa2086e9a9cb0e95 WatchSource:0}: Error finding container 902419a68fdfbc5c3f6942c930833b68e061f24ecbf474d4aa2086e9a9cb0e95: Status 404 returned error can't find the container with id 902419a68fdfbc5c3f6942c930833b68e061f24ecbf474d4aa2086e9a9cb0e95 Nov 03 22:47:33.115886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.115840 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" event={"ID":"dac539b8-4b24-4c8e-ab9c-aabaa943b303","Type":"ContainerStarted","Data":"b0d1d7aaad50b2f7e6206f0b9ba54090d5a025ec795ceb109cd0c4ddfc0cb451"} Nov 03 22:47:33.116122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.115914 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" event={"ID":"dac539b8-4b24-4c8e-ab9c-aabaa943b303","Type":"ContainerStarted","Data":"902419a68fdfbc5c3f6942c930833b68e061f24ecbf474d4aa2086e9a9cb0e95"} Nov 03 22:47:33.116122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.115967 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:33.117952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.117919 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" event={"ID":"72d94103-8548-4474-93d1-bc5a003d53b4","Type":"ContainerStarted","Data":"ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6"} Nov 03 22:47:33.118067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.117978 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" event={"ID":"72d94103-8548-4474-93d1-bc5a003d53b4","Type":"ContainerStarted","Data":"646128cfd64a9eb450fb2552e8b63bcce1138182ee3acb75649f35f1545eb998"} Nov 03 22:47:33.118067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.118011 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:33.119719 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.119693 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" event={"ID":"05346c7e-1f40-4a9a-a077-f2bfcd29eece","Type":"ContainerDied","Data":"ddd2a921776c5c1101118ad9e2134dc571682edd37df47c678c7e73eac404c9b"} Nov 03 22:47:33.119881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.119736 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddd2a921776c5c1101118ad9e2134dc571682edd37df47c678c7e73eac404c9b" Nov 03 22:47:33.119881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.119843 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d" Nov 03 22:47:33.123865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.123838 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" event={"ID":"26c3efad-74fe-49aa-8544-6d651cd87953","Type":"ContainerStarted","Data":"8282643b04198b41f1f55fab28a6285b8195ec253272f7a6027e7f8c48789716"} Nov 03 22:47:33.123981 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.123890 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" event={"ID":"26c3efad-74fe-49aa-8544-6d651cd87953","Type":"ContainerStarted","Data":"4d792e58197dadeeabb43a57d68ae34907396892e5b4c0d4db0ffc5596283eb6"} Nov 03 22:47:33.225383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.225328 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" podStartSLOduration=2.225314833 podStartE2EDuration="2.225314833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:33.224966193 +0000 UTC m=+1433.847274502" watchObservedRunningTime="2026-11-03 22:47:33.225314833 +0000 UTC m=+1433.847623141" Nov 03 22:47:33.334397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.334280 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" podStartSLOduration=2.334266471 podStartE2EDuration="2.334266471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:33.333429259 +0000 UTC m=+1433.955737570" watchObservedRunningTime="2026-11-03 22:47:33.334266471 +0000 UTC m=+1433.956574779" Nov 03 22:47:33.504838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.500508 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" podStartSLOduration=3.500488627 podStartE2EDuration="3.500488627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:33.499225768 +0000 UTC m=+1434.121534076" watchObservedRunningTime="2026-11-03 22:47:33.500488627 +0000 UTC m=+1434.122796936" Nov 03 22:47:33.586961 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.583783 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-vhkwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:47:33.586961 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:33.583863 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 22:47:34.137118 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:34.137082 5272 generic.go:356] "Generic (PLEG): container finished" podID="26c3efad-74fe-49aa-8544-6d651cd87953" containerID="8282643b04198b41f1f55fab28a6285b8195ec253272f7a6027e7f8c48789716" exitCode=0 Nov 03 22:47:34.137648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:34.137171 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" event={"ID":"26c3efad-74fe-49aa-8544-6d651cd87953","Type":"ContainerDied","Data":"8282643b04198b41f1f55fab28a6285b8195ec253272f7a6027e7f8c48789716"} Nov 03 22:47:34.588385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:34.588352 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:34.588932 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:34.588913 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:34.589639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:34.589614 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:47:35.304948 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.304917 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:35.414229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.414123 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-xks-ca\") pod \"26c3efad-74fe-49aa-8544-6d651cd87953\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " Nov 03 22:47:35.414420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.414390 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-openshift-ca\") pod \"26c3efad-74fe-49aa-8544-6d651cd87953\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " Nov 03 22:47:35.414489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.414463 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c3efad-74fe-49aa-8544-6d651cd87953-kube-api-access-492c5\" (UniqueName: \"kubernetes.io/projected/26c3efad-74fe-49aa-8544-6d651cd87953-kube-api-access-492c5\") pod \"26c3efad-74fe-49aa-8544-6d651cd87953\" (UID: \"26c3efad-74fe-49aa-8544-6d651cd87953\") " Nov 03 22:47:35.414888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.414852 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-openshift-ca" pod "26c3efad-74fe-49aa-8544-6d651cd87953" (UID: "26c3efad-74fe-49aa-8544-6d651cd87953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:35.415168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.415146 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:35.417151 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.417126 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-xks-ca" pod "26c3efad-74fe-49aa-8544-6d651cd87953" (UID: "26c3efad-74fe-49aa-8544-6d651cd87953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:35.417267 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.417200 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c3efad-74fe-49aa-8544-6d651cd87953-kube-api-access-492c5" pod "26c3efad-74fe-49aa-8544-6d651cd87953" (UID: "26c3efad-74fe-49aa-8544-6d651cd87953"). InnerVolumeSpecName "kube-api-access-492c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:35.553354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.553093 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492c5\" (UniqueName: \"kubernetes.io/projected/26c3efad-74fe-49aa-8544-6d651cd87953-kube-api-access-492c5\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:35.553354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:35.553137 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c3efad-74fe-49aa-8544-6d651cd87953-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:36.155672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:36.155643 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" Nov 03 22:47:36.155893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:36.155679 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2" event={"ID":"26c3efad-74fe-49aa-8544-6d651cd87953","Type":"ContainerDied","Data":"4d792e58197dadeeabb43a57d68ae34907396892e5b4c0d4db0ffc5596283eb6"} Nov 03 22:47:36.155893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:36.155727 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d792e58197dadeeabb43a57d68ae34907396892e5b4c0d4db0ffc5596283eb6" Nov 03 22:47:39.047391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:39.047355 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s"] Nov 03 22:47:39.093742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:39.093708 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1402d092-vg92s"] Nov 03 22:47:39.494912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:39.494835 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f93f005-86fd-4354-aca2-bc2a792a1556" path="/var/lib/kubelet/pods/3f93f005-86fd-4354-aca2-bc2a792a1556/volumes" Nov 03 22:47:41.550835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:41.550788 5272 scope.go:122] "RemoveContainer" containerID="450f423b60274b6b1bb43fdc00692a39c102d9cbea2c7c885337f77e5893056f" Nov 03 22:47:41.561762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:41.561740 5272 scope.go:122] "RemoveContainer" containerID="e9ce9402eb212f550972bbc4fca4291972ca715918da63a5e7889d2afb3dcc2b" Nov 03 22:47:41.571391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:41.571375 5272 scope.go:122] "RemoveContainer" containerID="e0ea5742010ab75b9ab261908a4175a64020111acda5ec5bc99e1b49b736bc5a" Nov 03 22:47:43.047475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:43.047441 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv"] Nov 03 22:47:43.093502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:43.093476 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86"] Nov 03 22:47:43.098156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:43.098131 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fae3e05d-9n2nv"] Nov 03 22:47:43.101941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:43.101919 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3783bd33-95m86"] Nov 03 22:47:43.506443 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:43.506356 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5886d361-f2f5-4d70-997a-5c496c8f0e1b" path="/var/lib/kubelet/pods/5886d361-f2f5-4d70-997a-5c496c8f0e1b/volumes" Nov 03 22:47:43.506931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:43.506908 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc84625-017e-4cdc-aec8-cf6ee28c811e" path="/var/lib/kubelet/pods/dfc84625-017e-4cdc-aec8-cf6ee28c811e/volumes" Nov 03 22:47:44.141062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:44.141026 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 22:47:44.141425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:44.141270 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 22:47:49.046674 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:49.046634 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs"] Nov 03 22:47:49.093120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:49.093088 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15edaa-ckmbs"] Nov 03 22:47:49.495224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:49.495128 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3032e8a-f500-4fc8-90ae-b95156531a81" path="/var/lib/kubelet/pods/b3032e8a-f500-4fc8-90ae-b95156531a81/volumes" Nov 03 22:47:51.047224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:51.047142 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr"] Nov 03 22:47:51.093258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:51.093223 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca8639-vl9pr"] Nov 03 22:47:51.495203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:51.495118 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513dbb9a-7533-4906-b95c-a12702e98ff1" path="/var/lib/kubelet/pods/513dbb9a-7533-4906-b95c-a12702e98ff1/volumes" Nov 03 22:47:54.663049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:54.663009 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:47:54.672446 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:47:54.672407 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:47:57.056827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:57.056397 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs"] Nov 03 22:47:57.093186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:57.093146 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df037bec-hkgqs"] Nov 03 22:47:57.523323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:57.523199 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7595d982-c998-4242-a084-3b3ad13b4c49" path="/var/lib/kubelet/pods/7595d982-c998-4242-a084-3b3ad13b4c49/volumes" Nov 03 22:47:58.338954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:58.338917 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rlkdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:47:58.339367 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:58.338989 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rlkdl" podUID="ed051394-e221-4f2a-bf68-988385bea856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:59.072188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:59.072129 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5"] Nov 03 22:47:59.093386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:59.093349 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043c526f-n8mw5"] Nov 03 22:47:59.496987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:47:59.496911 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a427e0-1ec9-4326-98fd-643a25fd28ac" path="/var/lib/kubelet/pods/d4a427e0-1ec9-4326-98fd-643a25fd28ac/volumes" Nov 03 22:48:00.071429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:00.071382 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m"] Nov 03 22:48:00.093244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:00.093207 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673aded6-b2m2m"] Nov 03 22:48:01.495855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:01.495814 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6713c7e9-c02e-4833-a9a3-d3646d01ea71" path="/var/lib/kubelet/pods/6713c7e9-c02e-4833-a9a3-d3646d01ea71/volumes" Nov 03 22:48:04.687822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:04.687602 5272 patch_prober.go:29] interesting pod/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:48:04.687822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:04.687677 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:48:06.534724 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:06.534692 5272 generic.go:356] "Generic (PLEG): container finished" podID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerID="a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa" exitCode=0 Nov 03 22:48:06.535203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:06.534763 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" event={"ID":"03aadb43-1ccb-4185-af97-9b69e3ee460b","Type":"ContainerDied","Data":"a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa"} Nov 03 22:48:06.535250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:06.535239 5272 scope.go:122] "RemoveContainer" containerID="a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa" Nov 03 22:48:06.536777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:06.536748 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:06.536859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:06.536838 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:06.536898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:06.536872 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:07.576233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:07.576187 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" event={"ID":"03aadb43-1ccb-4185-af97-9b69e3ee460b","Type":"ContainerStarted","Data":"96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284"} Nov 03 22:48:07.721892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:07.721855 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:48:07.722785 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:07.722171 5272 patch_prober.go:29] interesting pod/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:48:07.722785 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:07.722227 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 22:48:08.578746 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.578705 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp"] Nov 03 22:48:08.579990 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.579966 5272 cpu_manager.go:414] "containerMap: removing container" podUID="05346c7e-1f40-4a9a-a077-f2bfcd29eece" containerName="revoke-keys" Nov 03 22:48:08.580113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.580103 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05346c7e-1f40-4a9a-a077-f2bfcd29eece" containerName="revoke-keys" Nov 03 22:48:08.580175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.580168 5272 cpu_manager.go:414] "containerMap: removing container" podUID="26c3efad-74fe-49aa-8544-6d651cd87953" containerName="revoke-keys" Nov 03 22:48:08.580229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.580222 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c3efad-74fe-49aa-8544-6d651cd87953" containerName="revoke-keys" Nov 03 22:48:08.580460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.580450 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="05346c7e-1f40-4a9a-a077-f2bfcd29eece" containerName="revoke-keys" Nov 03 22:48:08.580529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.580522 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c3efad-74fe-49aa-8544-6d651cd87953" containerName="revoke-keys" Nov 03 22:48:08.584343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.584322 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.685943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.685910 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp"] Nov 03 22:48:08.692028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.691983 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.692190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.692030 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.692190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.692138 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.692337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.692219 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.692337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.692257 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.692486 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.692370 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.692486 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.692476 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.692795 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.692770 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.693375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.693315 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4qf\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-kube-api-access-9d4qf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.722271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.721361 5272 patch_prober.go:29] interesting pod/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:48:08.722271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.721419 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 22:48:08.795264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795230 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795299 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795369 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795408 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4qf\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-kube-api-access-9d4qf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795586 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795619 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795687 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795744 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795721 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.795794 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.795778 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.796030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.796012 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.796420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.796233 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.796420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.796268 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.796673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.796630 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.796939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.796915 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.797841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.797819 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.798308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.798284 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.870875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.870750 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4qf\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-kube-api-access-9d4qf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.871916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.871889 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xmfvp\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:08.909500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:08.909437 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:09.378846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.378796 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp"] Nov 03 22:48:09.382775 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:48:09.382746 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1b588c_ba6c_4546_bd6f_b71d7efacdc7.slice/crio-9c71df26bbf4911454740020987e0e8145eda1d09a582f1d6c1f309eeaa5f1a2 WatchSource:0}: Error finding container 9c71df26bbf4911454740020987e0e8145eda1d09a582f1d6c1f309eeaa5f1a2: Status 404 returned error can't find the container with id 9c71df26bbf4911454740020987e0e8145eda1d09a582f1d6c1f309eeaa5f1a2 Nov 03 22:48:09.386164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.386125 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.386266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.386198 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.386266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.386234 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:09.695936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.683734 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" event={"ID":"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7","Type":"ContainerStarted","Data":"a39407dfa7291dc2cc6b0286e871d6bd7374b724a8b5b5d784fd3ce11deaac25"} Nov 03 22:48:09.695936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.683819 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" event={"ID":"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7","Type":"ContainerStarted","Data":"9c71df26bbf4911454740020987e0e8145eda1d09a582f1d6c1f309eeaa5f1a2"} Nov 03 22:48:09.721425 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.721391 5272 patch_prober.go:29] interesting pod/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:48:09.721589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.721449 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 22:48:09.808160 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.808100 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podStartSLOduration=1.808081707 podStartE2EDuration="1.808081707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:09.8048938 +0000 UTC m=+1470.427202108" watchObservedRunningTime="2026-11-03 22:48:09.808081707 +0000 UTC m=+1470.430390016" Nov 03 22:48:09.910452 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:09.910419 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:10.775836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:10.775758 5272 patch_prober.go:29] interesting pod/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:48:10.808556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:10.808512 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 22:48:10.911297 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:10.911253 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:48:10.911504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:10.911304 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:48:11.721545 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:11.721505 5272 patch_prober.go:29] interesting pod/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:48:11.721845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:11.721602 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 22:48:11.910682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:11.910645 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:48:11.911233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:11.910691 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:48:12.721672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:12.721630 5272 patch_prober.go:29] interesting pod/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:48:12.721920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:12.721694 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 03 22:48:12.909821 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:12.909772 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:48:12.910091 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:12.909848 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:48:13.729548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:13.729512 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:48:13.731473 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:13.731447 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:48:13.732678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:13.732652 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:48:13.921829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:13.912292 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:48:13.921829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:13.912352 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:48:14.219248 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:14.219214 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n"] Nov 03 22:48:14.225835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:14.225787 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm"] Nov 03 22:48:14.233530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:14.233497 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8230bae9-td24n"] Nov 03 22:48:14.240822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:14.240772 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7370ae43-dzlpm"] Nov 03 22:48:14.909920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:14.909881 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:48:14.910592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:14.910560 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:48:15.495202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:15.495166 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f99857d-6f80-421c-b70d-5efb80a73712" path="/var/lib/kubelet/pods/8f99857d-6f80-421c-b70d-5efb80a73712/volumes" Nov 03 22:48:15.496431 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:15.496400 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc" path="/var/lib/kubelet/pods/ca1a2bd2-a8da-439c-b11c-d9cf1cc52ffc/volumes" Nov 03 22:48:15.910675 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:15.910639 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:48:15.911124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:15.910694 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:48:16.450416 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.450380 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5"] Nov 03 22:48:16.454796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.454774 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.509541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.509503 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5"] Nov 03 22:48:16.520754 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.520725 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:48:16.521031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.521009 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:48:16.523047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.523022 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcjd\" (UniqueName: \"kubernetes.io/projected/963f88ba-02ca-4801-93b2-533358357e8a-kube-api-access-8jcjd\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.523159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.523096 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.523222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.523184 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.548887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.548850 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:48:16.549041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.548921 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:48:16.625273 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.625243 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcjd\" (UniqueName: \"kubernetes.io/projected/963f88ba-02ca-4801-93b2-533358357e8a-kube-api-access-8jcjd\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.625460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.625308 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.625460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.625337 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.626165 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.626141 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.627775 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.627748 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.683212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.683177 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcjd\" (UniqueName: \"kubernetes.io/projected/963f88ba-02ca-4801-93b2-533358357e8a-kube-api-access-8jcjd\") pod \"maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.769917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.769827 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:16.909984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.909945 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 22:48:16.910187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:16.910023 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 22:48:17.160058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:17.160009 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5"] Nov 03 22:48:17.165131 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:48:17.165083 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963f88ba_02ca_4801_93b2_533358357e8a.slice/crio-bdc29fda9ddbb63bc53a1d749becac86f70f94ffa8c93f5d11a5f144bc8cb896 WatchSource:0}: Error finding container bdc29fda9ddbb63bc53a1d749becac86f70f94ffa8c93f5d11a5f144bc8cb896: Status 404 returned error can't find the container with id bdc29fda9ddbb63bc53a1d749becac86f70f94ffa8c93f5d11a5f144bc8cb896 Nov 03 22:48:17.957120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:17.957027 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" event={"ID":"963f88ba-02ca-4801-93b2-533358357e8a","Type":"ContainerStarted","Data":"33f3e4b3de5bdb7717f87646d147c66c4ec16e32b58aabd81f50a3eea6f6045e"} Nov 03 22:48:17.957120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:17.957092 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" event={"ID":"963f88ba-02ca-4801-93b2-533358357e8a","Type":"ContainerStarted","Data":"bdc29fda9ddbb63bc53a1d749becac86f70f94ffa8c93f5d11a5f144bc8cb896"} Nov 03 22:48:18.057337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.057299 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv"] Nov 03 22:48:18.061330 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.061302 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f08497c-dntnv"] Nov 03 22:48:18.092080 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.092028 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" podStartSLOduration=2.092014592 podStartE2EDuration="2.092014592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:18.091299423 +0000 UTC m=+1478.713607732" watchObservedRunningTime="2026-11-03 22:48:18.092014592 +0000 UTC m=+1478.714322964" Nov 03 22:48:18.636364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.636326 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:18.636882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.636850 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:18.637825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.637783 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 22:48:18.969893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.969787 5272 generic.go:356] "Generic (PLEG): container finished" podID="963f88ba-02ca-4801-93b2-533358357e8a" containerID="33f3e4b3de5bdb7717f87646d147c66c4ec16e32b58aabd81f50a3eea6f6045e" exitCode=0 Nov 03 22:48:18.970140 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:18.970103 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" event={"ID":"963f88ba-02ca-4801-93b2-533358357e8a","Type":"ContainerDied","Data":"33f3e4b3de5bdb7717f87646d147c66c4ec16e32b58aabd81f50a3eea6f6045e"} Nov 03 22:48:19.047530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:19.047492 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf"] Nov 03 22:48:19.051759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:19.051733 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccd515-jpdwf"] Nov 03 22:48:19.496345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:19.496301 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="410d2ee2-649b-42c4-805e-e6e50d1a7968" path="/var/lib/kubelet/pods/410d2ee2-649b-42c4-805e-e6e50d1a7968/volumes" Nov 03 22:48:19.496855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:19.496832 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f1f1e7-665a-4a74-9a01-929c65938114" path="/var/lib/kubelet/pods/80f1f1e7-665a-4a74-9a01-929c65938114/volumes" Nov 03 22:48:20.087368 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.087332 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w"] Nov 03 22:48:20.123923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.123883 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dec9a1-vm27w"] Nov 03 22:48:20.280618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.280588 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:20.403757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.403724 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963f88ba-02ca-4801-93b2-533358357e8a-kube-api-access-8jcjd\" (UniqueName: \"kubernetes.io/projected/963f88ba-02ca-4801-93b2-533358357e8a-kube-api-access-8jcjd\") pod \"963f88ba-02ca-4801-93b2-533358357e8a\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " Nov 03 22:48:20.403938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.403820 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-openshift-ca\") pod \"963f88ba-02ca-4801-93b2-533358357e8a\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " Nov 03 22:48:20.403979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.403944 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-xks-ca\") pod \"963f88ba-02ca-4801-93b2-533358357e8a\" (UID: \"963f88ba-02ca-4801-93b2-533358357e8a\") " Nov 03 22:48:20.404225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.404190 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-openshift-ca" pod "963f88ba-02ca-4801-93b2-533358357e8a" (UID: "963f88ba-02ca-4801-93b2-533358357e8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:20.404748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.404722 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:20.406193 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.406165 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-xks-ca" pod "963f88ba-02ca-4801-93b2-533358357e8a" (UID: "963f88ba-02ca-4801-93b2-533358357e8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:20.406496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.406473 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963f88ba-02ca-4801-93b2-533358357e8a-kube-api-access-8jcjd" pod "963f88ba-02ca-4801-93b2-533358357e8a" (UID: "963f88ba-02ca-4801-93b2-533358357e8a"). InnerVolumeSpecName "kube-api-access-8jcjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:20.506003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.505919 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jcjd\" (UniqueName: \"kubernetes.io/projected/963f88ba-02ca-4801-93b2-533358357e8a-kube-api-access-8jcjd\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:20.506003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.505948 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963f88ba-02ca-4801-93b2-533358357e8a-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:20.986830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.986762 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" Nov 03 22:48:20.986830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.986795 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5" event={"ID":"963f88ba-02ca-4801-93b2-533358357e8a","Type":"ContainerDied","Data":"bdc29fda9ddbb63bc53a1d749becac86f70f94ffa8c93f5d11a5f144bc8cb896"} Nov 03 22:48:20.987086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:20.986872 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc29fda9ddbb63bc53a1d749becac86f70f94ffa8c93f5d11a5f144bc8cb896" Nov 03 22:48:21.495480 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:21.495445 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb58de0e-903a-4dc0-b11b-c39bbafff353" path="/var/lib/kubelet/pods/eb58de0e-903a-4dc0-b11b-c39bbafff353/volumes" Nov 03 22:48:22.048400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:22.048364 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45"] Nov 03 22:48:22.094747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:22.094713 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e0f210-whm45"] Nov 03 22:48:23.495311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:23.495268 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9ba3d8-f5b1-4253-88b5-5bd31da31100" path="/var/lib/kubelet/pods/0c9ba3d8-f5b1-4253-88b5-5bd31da31100/volumes" Nov 03 22:48:24.733105 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:48:24.733070 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:48:26.050292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:26.050262 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz"] Nov 03 22:48:26.098151 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:26.098105 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dcee50e-xm9bz"] Nov 03 22:48:27.495907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:27.495875 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5509e52-244a-4f7c-a0c4-7313e566acee" path="/var/lib/kubelet/pods/d5509e52-244a-4f7c-a0c4-7313e566acee/volumes" Nov 03 22:48:28.806002 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:28.804181 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2"] Nov 03 22:48:33.072962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:33.072888 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg"] Nov 03 22:48:33.077226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:33.077193 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2a8438-s85hg"] Nov 03 22:48:33.556877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:33.550435 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45a9146-b621-4b7a-9522-3f5b65513845" path="/var/lib/kubelet/pods/c45a9146-b621-4b7a-9522-3f5b65513845/volumes" Nov 03 22:48:33.805517 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:33.805473 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" podUID="01619c55-1af7-4068-8cbf-8ffd9da02445" containerName="payload-pre-processing" containerID="cri-o://b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170" gracePeriod=25 Nov 03 22:48:34.125416 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.125374 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:48:34.192221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.192189 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01619c55-1af7-4068-8cbf-8ffd9da02445-kube-api-access-sqbtw\" (UniqueName: \"kubernetes.io/projected/01619c55-1af7-4068-8cbf-8ffd9da02445-kube-api-access-sqbtw\") pod \"01619c55-1af7-4068-8cbf-8ffd9da02445\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " Nov 03 22:48:34.192484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.192466 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01619c55-1af7-4068-8cbf-8ffd9da02445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01619c55-1af7-4068-8cbf-8ffd9da02445-plugins-config-volume\") pod \"01619c55-1af7-4068-8cbf-8ffd9da02445\" (UID: \"01619c55-1af7-4068-8cbf-8ffd9da02445\") " Nov 03 22:48:34.192855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.192826 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01619c55-1af7-4068-8cbf-8ffd9da02445-plugins-config-volume" pod "01619c55-1af7-4068-8cbf-8ffd9da02445" (UID: "01619c55-1af7-4068-8cbf-8ffd9da02445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:34.193099 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.193080 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01619c55-1af7-4068-8cbf-8ffd9da02445-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:34.194373 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.194354 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01619c55-1af7-4068-8cbf-8ffd9da02445-kube-api-access-sqbtw" pod "01619c55-1af7-4068-8cbf-8ffd9da02445" (UID: "01619c55-1af7-4068-8cbf-8ffd9da02445"). InnerVolumeSpecName "kube-api-access-sqbtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:34.215656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.215631 5272 generic.go:356] "Generic (PLEG): container finished" podID="01619c55-1af7-4068-8cbf-8ffd9da02445" containerID="b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170" exitCode=0 Nov 03 22:48:34.215755 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.215672 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" Nov 03 22:48:34.215755 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.215720 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" event={"ID":"01619c55-1af7-4068-8cbf-8ffd9da02445","Type":"ContainerDied","Data":"b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170"} Nov 03 22:48:34.215860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.215819 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2" event={"ID":"01619c55-1af7-4068-8cbf-8ffd9da02445","Type":"ContainerDied","Data":"dbd095bd29437c58261c9f17a9f2c67bfc9ee663fd9a1834698ac61fe9153462"} Nov 03 22:48:34.215905 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.215854 5272 scope.go:122] "RemoveContainer" containerID="b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170" Nov 03 22:48:34.228266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.228245 5272 scope.go:122] "RemoveContainer" containerID="b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170" Nov 03 22:48:34.228585 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:48:34.228565 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170\": container with ID starting with b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170 not found: ID does not exist" containerID="b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170" Nov 03 22:48:34.228660 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.228592 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170"} err="failed to get container status \"b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170\": rpc error: code = NotFound desc = could not find container \"b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170\": container with ID starting with b51a6aee68e462eadeb5a8e115e46583a1cb9696c93a5b1a59f56aed110a1170 not found: ID does not exist" Nov 03 22:48:34.294258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.294227 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbtw\" (UniqueName: \"kubernetes.io/projected/01619c55-1af7-4068-8cbf-8ffd9da02445-kube-api-access-sqbtw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:34.365982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.365861 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2"] Nov 03 22:48:34.374707 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:34.374678 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7a91ca4-65b8c9ccf-h87z2"] Nov 03 22:48:35.494583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:35.494551 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01619c55-1af7-4068-8cbf-8ffd9da02445" path="/var/lib/kubelet/pods/01619c55-1af7-4068-8cbf-8ffd9da02445/volumes" Nov 03 22:48:36.007544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:36.007499 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc"] Nov 03 22:48:36.007780 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:36.007750 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" containerID="cri-o://85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca" gracePeriod=30 Nov 03 22:48:36.053734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:36.053704 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft"] Nov 03 22:48:36.053940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:36.053918 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" containerID="cri-o://425589bbab578f19f09ca9af43ff6a4878296347833a0ac2d8db14e570f5575e" gracePeriod=30 Nov 03 22:48:39.342968 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:39.342929 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:48:39.343454 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:39.342995 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:48:40.347143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:40.347107 5272 generic.go:356] "Generic (PLEG): container finished" podID="f32b4f71-9f49-4f92-b766-88ce76449123" containerID="5aab74f971329a16fb7ee278f4f3db2bfbea96d0eddcb514313eedc9eab7986d" exitCode=0 Nov 03 22:48:40.347578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:40.347180 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" event={"ID":"f32b4f71-9f49-4f92-b766-88ce76449123","Type":"ContainerDied","Data":"5aab74f971329a16fb7ee278f4f3db2bfbea96d0eddcb514313eedc9eab7986d"} Nov 03 22:48:40.347645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:40.347634 5272 scope.go:122] "RemoveContainer" containerID="5aab74f971329a16fb7ee278f4f3db2bfbea96d0eddcb514313eedc9eab7986d" Nov 03 22:48:40.349283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:40.349251 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:40.349408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:40.349336 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:40.349408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:40.349385 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:48:41.092695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.092664 5272 patch_prober.go:29] interesting pod/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:48:41.092893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.092725 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:48:41.357039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.357007 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:48:41.358022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.357998 5272 generic.go:356] "Generic (PLEG): container finished" podID="9a53f698-8f33-40d5-a91b-006f922f4427" containerID="85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca" exitCode=0 Nov 03 22:48:41.358120 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.358072 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" event={"ID":"9a53f698-8f33-40d5-a91b-006f922f4427","Type":"ContainerDied","Data":"85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca"} Nov 03 22:48:41.358176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.358132 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" event={"ID":"9a53f698-8f33-40d5-a91b-006f922f4427","Type":"ContainerDied","Data":"498ac1bab517894674d90487d0cbe255111fbeba203bf7960136a6e1e3515547"} Nov 03 22:48:41.358176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.358167 5272 scope.go:122] "RemoveContainer" containerID="85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca" Nov 03 22:48:41.360413 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.360379 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" event={"ID":"f32b4f71-9f49-4f92-b766-88ce76449123","Type":"ContainerStarted","Data":"a32dc59afb658b1e58bc6e3028c200ba19fd9e0a0361024fd4794e6e92b9900f"} Nov 03 22:48:41.362291 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.362269 5272 generic.go:356] "Generic (PLEG): container finished" podID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerID="425589bbab578f19f09ca9af43ff6a4878296347833a0ac2d8db14e570f5575e" exitCode=0 Nov 03 22:48:41.362407 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.362308 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" event={"ID":"333beb40-80d8-41d6-9d49-1308c7a6df9e","Type":"ContainerDied","Data":"425589bbab578f19f09ca9af43ff6a4878296347833a0ac2d8db14e570f5575e"} Nov 03 22:48:41.374659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.374630 5272 scope.go:122] "RemoveContainer" containerID="85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca" Nov 03 22:48:41.375022 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:48:41.374991 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca\": container with ID starting with 85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca not found: ID does not exist" containerID="85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca" Nov 03 22:48:41.375137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.375032 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca"} err="failed to get container status \"85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca\": rpc error: code = NotFound desc = could not find container \"85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca\": container with ID starting with 85685ef84e8259b25af9a58e4c99005c60ed4bc407bfa44b62fb05c987a831ca not found: ID does not exist" Nov 03 22:48:41.386282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386252 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-envoy\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.386394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386344 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-data\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.386394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386384 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-kube-api-access-w5gsj\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-kube-api-access-w5gsj\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.386510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386441 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-certs\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.386574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386552 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-istio-token\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.387239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386723 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-credential-socket\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.387239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386784 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-data" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.387239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.386951 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-certs" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.387239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.387060 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-credential-socket" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.387239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.387124 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-socket\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.387576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.387252 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a53f698-8f33-40d5-a91b-006f922f4427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f698-8f33-40d5-a91b-006f922f4427-istiod-ca-cert\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.387576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.387313 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a53f698-8f33-40d5-a91b-006f922f4427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f698-8f33-40d5-a91b-006f922f4427-istio-podinfo\") pod \"9a53f698-8f33-40d5-a91b-006f922f4427\" (UID: \"9a53f698-8f33-40d5-a91b-006f922f4427\") " Nov 03 22:48:41.388059 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.388035 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.388170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.388062 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.388170 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.388076 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.388500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.388469 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-envoy" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.389190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.389165 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a53f698-8f33-40d5-a91b-006f922f4427-istiod-ca-cert" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:41.389321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.389300 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-kube-api-access-w5gsj" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "kube-api-access-w5gsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:41.389668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.389644 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-socket" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.389865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.389845 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-istio-token" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:41.390679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.390661 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a53f698-8f33-40d5-a91b-006f922f4427-istio-podinfo" pod "9a53f698-8f33-40d5-a91b-006f922f4427" (UID: "9a53f698-8f33-40d5-a91b-006f922f4427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:48:41.399158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.399139 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:48:41.442965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.442935 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:48:41.443130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.443110 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:48:41.443194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.443156 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 22:48:41.511201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.511158 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.511201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.511204 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5gsj\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-kube-api-access-w5gsj\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.511535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.511220 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53f698-8f33-40d5-a91b-006f922f4427-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.511535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.511235 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53f698-8f33-40d5-a91b-006f922f4427-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.511535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.511256 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53f698-8f33-40d5-a91b-006f922f4427-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.511535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.511272 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53f698-8f33-40d5-a91b-006f922f4427-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.612030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.611978 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-kube-api-access-sbsfw\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-kube-api-access-sbsfw\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.612356 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.612333 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-envoy\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.612527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.612507 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-data\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.612702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.612676 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-token\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-token\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.612869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.612769 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-certs\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.612950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.612870 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-credential-socket\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.612950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.612899 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333beb40-80d8-41d6-9d49-1308c7a6df9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333beb40-80d8-41d6-9d49-1308c7a6df9e-istiod-ca-cert\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.613060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.612950 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-socket\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.613116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.613062 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-podinfo\") pod \"333beb40-80d8-41d6-9d49-1308c7a6df9e\" (UID: \"333beb40-80d8-41d6-9d49-1308c7a6df9e\") " Nov 03 22:48:41.616153 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.616102 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-certs" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.616290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.616186 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-data" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.616420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.616400 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-socket" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.616757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.616599 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-credential-socket" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.616757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.616711 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333beb40-80d8-41d6-9d49-1308c7a6df9e-istiod-ca-cert" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:41.617013 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.616993 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-podinfo" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:48:41.618459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.618436 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-token" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:41.618659 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.618636 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-envoy" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:41.618703 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.618659 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-kube-api-access-sbsfw" pod "333beb40-80d8-41d6-9d49-1308c7a6df9e" (UID: "333beb40-80d8-41d6-9d49-1308c7a6df9e"). InnerVolumeSpecName "kube-api-access-sbsfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:41.632592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.632131 5272 scope.go:122] "RemoveContainer" containerID="0757f868ade9558db5485ecf9ab5a53da91acb38c456b0988bb4d764765e213c" Nov 03 22:48:41.673711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.673676 5272 scope.go:122] "RemoveContainer" containerID="6d2c9593e617014ed2837598ddf6101e6f20c7e5bfe47411494ccf10800a9406" Nov 03 22:48:41.687211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.687189 5272 scope.go:122] "RemoveContainer" containerID="9d136a0cff3aaf24a5dc984f52155c3217bd4d35124cbbe8ad200a54f581971d" Nov 03 22:48:41.700867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.700845 5272 scope.go:122] "RemoveContainer" containerID="8cb244e907d698cb1518f13d350bcf8a61c0bb74956f34facf1d1e19a1c8b702" Nov 03 22:48:41.724479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.724444 5272 scope.go:122] "RemoveContainer" containerID="f6952af59f1ab021f29cdb091004a195b40d940bb7f20144ab5ec0a041d9c43a" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727120 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727149 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbsfw\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-kube-api-access-sbsfw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727167 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727182 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727198 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/333beb40-80d8-41d6-9d49-1308c7a6df9e-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727214 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727231 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727247 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/333beb40-80d8-41d6-9d49-1308c7a6df9e-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.727277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.727262 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/333beb40-80d8-41d6-9d49-1308c7a6df9e-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:41.737020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.736995 5272 scope.go:122] "RemoveContainer" containerID="65d0fa0f73bff0797c941ac296532bd63c6261cf1b4b262e662cb0649f6ace46" Nov 03 22:48:41.770417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:41.770275 5272 scope.go:122] "RemoveContainer" containerID="68f4fc64f1f0db1de4b856c162fcbde55f4a92e3d4af2dba579728de852aac35" Nov 03 22:48:42.070589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.070546 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn"] Nov 03 22:48:42.074600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.074539 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c664cc-969jn"] Nov 03 22:48:42.377797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.377709 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" Nov 03 22:48:42.377797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.377715 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft" event={"ID":"333beb40-80d8-41d6-9d49-1308c7a6df9e","Type":"ContainerDied","Data":"7ec390069b6fb47943c8a32b9935de6b6a1a127096f9b0ce897e38723b5e40af"} Nov 03 22:48:42.378395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.377890 5272 scope.go:122] "RemoveContainer" containerID="425589bbab578f19f09ca9af43ff6a4878296347833a0ac2d8db14e570f5575e" Nov 03 22:48:42.378395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.378155 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc" Nov 03 22:48:42.459844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.456294 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:48:42.460228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.460187 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 22:48:42.536733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.536694 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft"] Nov 03 22:48:42.570789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.570754 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-vw6ft"] Nov 03 22:48:42.714006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.713908 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc"] Nov 03 22:48:42.720244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:42.720216 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-961054-openshift-default-5d5665c66f-w8xsc"] Nov 03 22:48:43.466828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:43.466244 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:48:43.466828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:43.466315 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 22:48:43.494445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:43.494409 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" path="/var/lib/kubelet/pods/333beb40-80d8-41d6-9d49-1308c7a6df9e/volumes" Nov 03 22:48:43.494926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:43.494911 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75439837-68b1-4d0b-8fbc-df6bf3108412" path="/var/lib/kubelet/pods/75439837-68b1-4d0b-8fbc-df6bf3108412/volumes" Nov 03 22:48:43.495350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:43.495338 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" path="/var/lib/kubelet/pods/9a53f698-8f33-40d5-a91b-006f922f4427/volumes" Nov 03 22:48:44.461828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:44.461145 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:48:44.461828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:44.461209 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 22:48:45.443830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:45.443304 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:48:45.443830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:45.443388 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 22:48:45.628877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:45.628837 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:48:46.462828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:46.447891 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:48:46.462828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:46.447959 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 22:48:47.447440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:47.447406 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:48:47.447846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:47.447821 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:48:47.448515 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:47.448496 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 22:48:54.502643 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:48:54.502604 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:48:56.046567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:56.046528 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df"] Nov 03 22:48:56.093063 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:56.093031 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f94c37-sw2df"] Nov 03 22:48:57.495951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:57.495909 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4a8d65-e57d-42e7-a447-3ae0504e4328" path="/var/lib/kubelet/pods/8e4a8d65-e57d-42e7-a447-3ae0504e4328/volumes" Nov 03 22:48:59.056698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:59.056658 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk"] Nov 03 22:48:59.103108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:59.103075 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0440c5-f5sbk"] Nov 03 22:48:59.503559 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:48:59.503480 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33f928f-81a5-46e7-a636-2d210a75170f" path="/var/lib/kubelet/pods/c33f928f-81a5-46e7-a636-2d210a75170f/volumes" Nov 03 22:49:03.961717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.961669 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt"] Nov 03 22:49:03.962551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962529 5272 cpu_manager.go:414] "containerMap: removing container" podUID="01619c55-1af7-4068-8cbf-8ffd9da02445" containerName="payload-pre-processing" Nov 03 22:49:03.962597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962553 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01619c55-1af7-4068-8cbf-8ffd9da02445" containerName="payload-pre-processing" Nov 03 22:49:03.962597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962580 5272 cpu_manager.go:414] "containerMap: removing container" podUID="963f88ba-02ca-4801-93b2-533358357e8a" containerName="revoke-keys" Nov 03 22:49:03.962597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962585 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963f88ba-02ca-4801-93b2-533358357e8a" containerName="revoke-keys" Nov 03 22:49:03.962597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962597 5272 cpu_manager.go:414] "containerMap: removing container" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" Nov 03 22:49:03.962726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962602 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" Nov 03 22:49:03.962726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962613 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" Nov 03 22:49:03.962726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962618 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" Nov 03 22:49:03.962726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962726 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="01619c55-1af7-4068-8cbf-8ffd9da02445" containerName="payload-pre-processing" Nov 03 22:49:03.962895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962739 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="963f88ba-02ca-4801-93b2-533358357e8a" containerName="revoke-keys" Nov 03 22:49:03.962895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962746 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="333beb40-80d8-41d6-9d49-1308c7a6df9e" containerName="istio-proxy" Nov 03 22:49:03.962895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.962753 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a53f698-8f33-40d5-a91b-006f922f4427" containerName="istio-proxy" Nov 03 22:49:03.966589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:03.966566 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.006721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.006675 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.006950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.006734 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-token\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.006950 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.006917 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-socket\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.007081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.006976 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pdw\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-kube-api-access-c9pdw\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.007145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.007075 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-podinfo\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.007237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.007220 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-credential-socket\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.007296 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.007249 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-certs\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.007351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.007311 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-data\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.007406 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.007362 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-envoy\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.017938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.017905 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fb4f0a-openshift-default-dockercfg-c6ctl\"" Nov 03 22:49:04.068323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.068290 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k"] Nov 03 22:49:04.072173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.072144 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ea7f22-v9b9k"] Nov 03 22:49:04.075707 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.075680 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt"] Nov 03 22:49:04.109286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109248 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109299 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-token\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109367 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-socket\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109416 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pdw\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-kube-api-access-c9pdw\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109510 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-podinfo\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109653 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-credential-socket\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109677 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-certs\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109737 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-data\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.109982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109786 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-envoy\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.110107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.109994 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-socket\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.110191 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.110162 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-credential-socket\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.110297 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.110221 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-certs\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.110692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.110665 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-data\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.111598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.111572 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.112855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.112817 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-envoy\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.113471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.113382 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-podinfo\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.256868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.248457 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-token\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.256868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.253875 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pdw\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-kube-api-access-c9pdw\") pod \"e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.300829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.286390 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:04.999274 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:04.999184 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt"] Nov 03 22:49:05.004587 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:05.004557 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade70f21_d0ee_4c05_aec9_602d5f1e22eb.slice/crio-692c6204e61a27a205c24354ec493cfb491ecf72be916d323634fdd9c5b34a1e WatchSource:0}: Error finding container 692c6204e61a27a205c24354ec493cfb491ecf72be916d323634fdd9c5b34a1e: Status 404 returned error can't find the container with id 692c6204e61a27a205c24354ec493cfb491ecf72be916d323634fdd9c5b34a1e Nov 03 22:49:05.007269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:05.007221 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:05.007358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:05.007305 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:05.007358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:05.007337 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:05.496362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:05.496330 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95" path="/var/lib/kubelet/pods/f043d407-a1dc-4fa0-8e9d-9e2d9d9b5f95/volumes" Nov 03 22:49:05.649311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:05.649274 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" event={"ID":"ade70f21-d0ee-4c05-aec9-602d5f1e22eb","Type":"ContainerStarted","Data":"c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1"} Nov 03 22:49:05.649487 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:05.649329 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" event={"ID":"ade70f21-d0ee-4c05-aec9-602d5f1e22eb","Type":"ContainerStarted","Data":"692c6204e61a27a205c24354ec493cfb491ecf72be916d323634fdd9c5b34a1e"} Nov 03 22:49:05.762101 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:05.761956 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podStartSLOduration=2.76193933 podStartE2EDuration="2.76193933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:05.761266426 +0000 UTC m=+1526.383574735" watchObservedRunningTime="2026-11-03 22:49:05.76193933 +0000 UTC m=+1526.384247639" Nov 03 22:49:06.287375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:06.287347 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:06.292463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:06.292437 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:06.657996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:06.657915 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:06.658913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:06.658893 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:49:09.831168 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:09.831130 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5"] Nov 03 22:49:09.838002 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:09.837976 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:09.893672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:09.893635 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:49:09.893916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:09.893720 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:49:09.947383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:09.947349 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:49:10.068184 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.001458 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:49:10.080630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.069623 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.080630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.069767 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.080630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.069923 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghbm\" (UniqueName: \"kubernetes.io/projected/8b844aaa-b202-44c2-acc8-3e004029bca0-kube-api-access-5ghbm\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.107088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.106993 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5"] Nov 03 22:49:10.178918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.178868 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.179102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.179027 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.179188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.179173 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghbm\" (UniqueName: \"kubernetes.io/projected/8b844aaa-b202-44c2-acc8-3e004029bca0-kube-api-access-5ghbm\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.180622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.180592 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.182346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.182320 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.237250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.237212 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghbm\" (UniqueName: \"kubernetes.io/projected/8b844aaa-b202-44c2-acc8-3e004029bca0-kube-api-access-5ghbm\") pod \"maas-api-revoke-sub-20a8c1c7-sgxt5\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.484312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.484211 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:10.889108 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:10.889078 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b844aaa_b202_44c2_acc8_3e004029bca0.slice/crio-5a5048fc4f7125b926628c4817714afa288f1c7e5807ce77fab74ad6f3a9a3a6 WatchSource:0}: Error finding container 5a5048fc4f7125b926628c4817714afa288f1c7e5807ce77fab74ad6f3a9a3a6: Status 404 returned error can't find the container with id 5a5048fc4f7125b926628c4817714afa288f1c7e5807ce77fab74ad6f3a9a3a6 Nov 03 22:49:10.900061 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:10.900036 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5"] Nov 03 22:49:11.712982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:11.712940 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" event={"ID":"8b844aaa-b202-44c2-acc8-3e004029bca0","Type":"ContainerStarted","Data":"da412c8f7c856588990f86684503f9b5406cda99febf8cc208904aa87bfa4e24"} Nov 03 22:49:11.713185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:11.713007 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" event={"ID":"8b844aaa-b202-44c2-acc8-3e004029bca0","Type":"ContainerStarted","Data":"5a5048fc4f7125b926628c4817714afa288f1c7e5807ce77fab74ad6f3a9a3a6"} Nov 03 22:49:11.823998 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:11.823943 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" podStartSLOduration=2.823928961 podStartE2EDuration="2.823928961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:11.822191164 +0000 UTC m=+1532.444499473" watchObservedRunningTime="2026-11-03 22:49:11.823928961 +0000 UTC m=+1532.446237268" Nov 03 22:49:12.103878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.102388 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg"] Nov 03 22:49:12.256900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.249186 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750d0d7-r7hcg"] Nov 03 22:49:12.674827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.673641 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229"] Nov 03 22:49:12.696826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.679910 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:12.735450 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.728217 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229"] Nov 03 22:49:12.837834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.835396 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-fb4f0a-dockercfg-xqxkk\"" Nov 03 22:49:12.837834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.835535 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fb4f0a\"" Nov 03 22:49:12.870870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.843895 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5ffd82-72be-42b9-be1b-b7ac56057509-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:12.870870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.844111 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57hf\" (UniqueName: \"kubernetes.io/projected/2f5ffd82-72be-42b9-be1b-b7ac56057509-kube-api-access-v57hf\") pod \"payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:12.940006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.939248 5272 generic.go:356] "Generic (PLEG): container finished" podID="8b844aaa-b202-44c2-acc8-3e004029bca0" containerID="da412c8f7c856588990f86684503f9b5406cda99febf8cc208904aa87bfa4e24" exitCode=0 Nov 03 22:49:12.940006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.939426 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" event={"ID":"8b844aaa-b202-44c2-acc8-3e004029bca0","Type":"ContainerDied","Data":"da412c8f7c856588990f86684503f9b5406cda99febf8cc208904aa87bfa4e24"} Nov 03 22:49:12.992864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.991999 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57hf\" (UniqueName: \"kubernetes.io/projected/2f5ffd82-72be-42b9-be1b-b7ac56057509-kube-api-access-v57hf\") pod \"payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:12.992864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.992252 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5ffd82-72be-42b9-be1b-b7ac56057509-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:12.994067 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:12.993887 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5ffd82-72be-42b9-be1b-b7ac56057509-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:13.062744 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.062710 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr"] Nov 03 22:49:13.201162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.198907 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57hf\" (UniqueName: \"kubernetes.io/projected/2f5ffd82-72be-42b9-be1b-b7ac56057509-kube-api-access-v57hf\") pod \"payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:13.206361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.206199 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr"] Nov 03 22:49:13.206361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.206326 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.296651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.296611 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:13.297547 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.297521 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.297620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.297572 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsf5\" (UniqueName: \"kubernetes.io/projected/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-kube-api-access-wjsf5\") pod \"payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.399185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.399148 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.399321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.399212 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsf5\" (UniqueName: \"kubernetes.io/projected/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-kube-api-access-wjsf5\") pod \"payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.400208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.400178 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.471051 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.470926 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsf5\" (UniqueName: \"kubernetes.io/projected/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-kube-api-access-wjsf5\") pod \"payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.496041 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.495998 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46" path="/var/lib/kubelet/pods/cfbf3d1a-9cef-434f-8dc1-cc39b74a1b46/volumes" Nov 03 22:49:13.566449 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.566409 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:13.704638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.704605 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229"] Nov 03 22:49:13.712114 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:13.712072 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5ffd82_72be_42b9_be1b_b7ac56057509.slice/crio-39c2990b872d724d08f1226f6a366243504868fb809a970da5c7a1e9833ab29f WatchSource:0}: Error finding container 39c2990b872d724d08f1226f6a366243504868fb809a970da5c7a1e9833ab29f: Status 404 returned error can't find the container with id 39c2990b872d724d08f1226f6a366243504868fb809a970da5c7a1e9833ab29f Nov 03 22:49:13.952077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.952042 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr"] Nov 03 22:49:13.952416 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.952386 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" event={"ID":"2f5ffd82-72be-42b9-be1b-b7ac56057509","Type":"ContainerStarted","Data":"c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3"} Nov 03 22:49:13.952504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.952450 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" event={"ID":"2f5ffd82-72be-42b9-be1b-b7ac56057509","Type":"ContainerStarted","Data":"39c2990b872d724d08f1226f6a366243504868fb809a970da5c7a1e9833ab29f"} Nov 03 22:49:13.952504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:13.952481 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:13.956148 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:13.956109 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee77aaff_2035_49ba_9d77_5f13a2ad4a30.slice/crio-00b27b5f6f04c66b8a9577033d2143db2434bcc8d1fbb48546e52cc379c06599 WatchSource:0}: Error finding container 00b27b5f6f04c66b8a9577033d2143db2434bcc8d1fbb48546e52cc379c06599: Status 404 returned error can't find the container with id 00b27b5f6f04c66b8a9577033d2143db2434bcc8d1fbb48546e52cc379c06599 Nov 03 22:49:14.052841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.052485 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl"] Nov 03 22:49:14.059250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.059211 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c2fd7b-skfdl"] Nov 03 22:49:14.066526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.066455 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" podStartSLOduration=2.066432046 podStartE2EDuration="2.066432046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:14.0632647 +0000 UTC m=+1534.685573008" watchObservedRunningTime="2026-11-03 22:49:14.066432046 +0000 UTC m=+1534.688740355" Nov 03 22:49:14.110112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.110079 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:14.211340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.211303 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-openshift-ca\") pod \"8b844aaa-b202-44c2-acc8-3e004029bca0\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " Nov 03 22:49:14.211732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.211406 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-xks-ca\") pod \"8b844aaa-b202-44c2-acc8-3e004029bca0\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " Nov 03 22:49:14.211732 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.211702 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-openshift-ca" pod "8b844aaa-b202-44c2-acc8-3e004029bca0" (UID: "8b844aaa-b202-44c2-acc8-3e004029bca0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:14.211867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.211776 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b844aaa-b202-44c2-acc8-3e004029bca0-kube-api-access-5ghbm\" (UniqueName: \"kubernetes.io/projected/8b844aaa-b202-44c2-acc8-3e004029bca0-kube-api-access-5ghbm\") pod \"8b844aaa-b202-44c2-acc8-3e004029bca0\" (UID: \"8b844aaa-b202-44c2-acc8-3e004029bca0\") " Nov 03 22:49:14.212511 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.212483 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:14.213778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.213753 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b844aaa-b202-44c2-acc8-3e004029bca0-kube-api-access-5ghbm" pod "8b844aaa-b202-44c2-acc8-3e004029bca0" (UID: "8b844aaa-b202-44c2-acc8-3e004029bca0"). InnerVolumeSpecName "kube-api-access-5ghbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:14.213928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.213903 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-xks-ca" pod "8b844aaa-b202-44c2-acc8-3e004029bca0" (UID: "8b844aaa-b202-44c2-acc8-3e004029bca0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:14.314256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.314208 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ghbm\" (UniqueName: \"kubernetes.io/projected/8b844aaa-b202-44c2-acc8-3e004029bca0-kube-api-access-5ghbm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:14.314256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.314249 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b844aaa-b202-44c2-acc8-3e004029bca0-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:15.005825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.999551 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" event={"ID":"ee77aaff-2035-49ba-9d77-5f13a2ad4a30","Type":"ContainerStarted","Data":"9f82c580ab7dceabb80966a7e19fc75db03b6477836fa883d2e4420ae02d5432"} Nov 03 22:49:15.005825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:14.999623 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" event={"ID":"ee77aaff-2035-49ba-9d77-5f13a2ad4a30","Type":"ContainerStarted","Data":"00b27b5f6f04c66b8a9577033d2143db2434bcc8d1fbb48546e52cc379c06599"} Nov 03 22:49:15.022875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:15.011988 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:15.050657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:15.048736 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" Nov 03 22:49:15.050657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:15.050074 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5" event={"ID":"8b844aaa-b202-44c2-acc8-3e004029bca0","Type":"ContainerDied","Data":"5a5048fc4f7125b926628c4817714afa288f1c7e5807ce77fab74ad6f3a9a3a6"} Nov 03 22:49:15.050657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:15.050203 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a5048fc4f7125b926628c4817714afa288f1c7e5807ce77fab74ad6f3a9a3a6" Nov 03 22:49:15.124402 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:15.124337 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" podStartSLOduration=3.124319812 podStartE2EDuration="3.124319812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:15.122666481 +0000 UTC m=+1535.744974791" watchObservedRunningTime="2026-11-03 22:49:15.124319812 +0000 UTC m=+1535.746628121" Nov 03 22:49:15.564824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:15.552210 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d9e954-4292-440d-866f-787832480959" path="/var/lib/kubelet/pods/b5d9e954-4292-440d-866f-787832480959/volumes" Nov 03 22:49:16.597825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:16.597193 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5"] Nov 03 22:49:16.630594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:16.630555 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7a91ca4-7lks5"] Nov 03 22:49:16.678511 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:16.677308 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v"] Nov 03 22:49:16.678511 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:16.678039 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" containerID="cri-o://96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284" gracePeriod=30 Nov 03 22:49:17.498506 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:17.498440 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963f88ba-02ca-4801-93b2-533358357e8a" path="/var/lib/kubelet/pods/963f88ba-02ca-4801-93b2-533358357e8a/volumes" Nov 03 22:49:18.133038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.133003 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml"] Nov 03 22:49:18.134057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.134035 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8b844aaa-b202-44c2-acc8-3e004029bca0" containerName="revoke-keys" Nov 03 22:49:18.134057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.134058 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b844aaa-b202-44c2-acc8-3e004029bca0" containerName="revoke-keys" Nov 03 22:49:18.134248 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.134232 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b844aaa-b202-44c2-acc8-3e004029bca0" containerName="revoke-keys" Nov 03 22:49:18.170770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.170736 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.215879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.208013 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-socket\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.216202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.216170 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-podinfo\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.249829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.216380 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-credential-socket\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.284028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.283949 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-token\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.291889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.280920 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-6a3169-openshift-default-dockercfg-ln6d2\"" Nov 03 22:49:18.291889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.284413 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d448175a-3e0e-4b5d-b786-010d6a58a1af-istiod-ca-cert\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.291889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.284504 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-envoy\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.291889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.284536 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdt7v\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-kube-api-access-bdt7v\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.291889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.284569 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-certs\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.291889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.284705 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-data\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390147 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-token\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390287 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d448175a-3e0e-4b5d-b786-010d6a58a1af-istiod-ca-cert\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390318 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-envoy\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390342 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdt7v\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-kube-api-access-bdt7v\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390374 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-certs\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390477 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-data\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390642 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-socket\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390675 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-podinfo\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.390700 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-credential-socket\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.391268 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-credential-socket\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.392175 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d448175a-3e0e-4b5d-b786-010d6a58a1af-istiod-ca-cert\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.412747 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-data\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.413445 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-certs\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.445032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.413670 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-socket\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.487034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.486990 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdt7v\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-kube-api-access-bdt7v\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.487217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.487081 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml"] Nov 03 22:49:18.487334 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.487309 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-podinfo\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.514153 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.512996 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-token\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.549471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.542874 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-envoy\") pod \"e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:18.731513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:18.731421 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:19.263613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:19.263581 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml"] Nov 03 22:49:19.267558 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:19.267530 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd448175a_3e0e_4b5d_b786_010d6a58a1af.slice/crio-b4f7288f9ca2f98e5fae24c2b72fa3229856d047c4d3ca196585f11e9e1ed92e WatchSource:0}: Error finding container b4f7288f9ca2f98e5fae24c2b72fa3229856d047c4d3ca196585f11e9e1ed92e: Status 404 returned error can't find the container with id b4f7288f9ca2f98e5fae24c2b72fa3229856d047c4d3ca196585f11e9e1ed92e Nov 03 22:49:19.270947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:19.270908 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:19.271056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:19.270984 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:19.271056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:19.271015 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:20.185942 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.185854 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerID="3fe0c16cc46fb6b435a46726669df7003222fc6b2f432d77b83a33f2d4078501" exitCode=0 Nov 03 22:49:20.186139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.185933 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" event={"ID":"1ed69d8c-da98-4c76-8d68-b475ebb41024","Type":"ContainerDied","Data":"3fe0c16cc46fb6b435a46726669df7003222fc6b2f432d77b83a33f2d4078501"} Nov 03 22:49:20.186392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.186376 5272 scope.go:122] "RemoveContainer" containerID="3fe0c16cc46fb6b435a46726669df7003222fc6b2f432d77b83a33f2d4078501" Nov 03 22:49:20.188087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.188065 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" event={"ID":"d448175a-3e0e-4b5d-b786-010d6a58a1af","Type":"ContainerStarted","Data":"f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb"} Nov 03 22:49:20.188219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.188104 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" event={"ID":"d448175a-3e0e-4b5d-b786-010d6a58a1af","Type":"ContainerStarted","Data":"b4f7288f9ca2f98e5fae24c2b72fa3229856d047c4d3ca196585f11e9e1ed92e"} Nov 03 22:49:20.188219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.188172 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:20.188303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.188257 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:20.188351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.188335 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:20.463709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.463609 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podStartSLOduration=2.463593613 podStartE2EDuration="2.463593613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:20.460619673 +0000 UTC m=+1541.082927983" watchObservedRunningTime="2026-11-03 22:49:20.463593613 +0000 UTC m=+1541.085901920" Nov 03 22:49:20.732771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.732688 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:20.738767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:20.738739 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:21.200721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:21.200686 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" event={"ID":"1ed69d8c-da98-4c76-8d68-b475ebb41024","Type":"ContainerStarted","Data":"36057d33e274c96702e6bc1046f8a14e6e4986145f6e33d9bb7944017dba842d"} Nov 03 22:49:21.201139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:21.201115 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:21.201904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:21.201884 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:49:21.762993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:21.759837 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:49:21.770440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:21.770399 5272 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:49:21.770620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:21.770468 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 22:49:22.158552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.158025 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:49:22.210789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.210750 5272 generic.go:356] "Generic (PLEG): container finished" podID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerID="96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284" exitCode=0 Nov 03 22:49:22.210979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.210824 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" Nov 03 22:49:22.210979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.210834 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" event={"ID":"03aadb43-1ccb-4185-af97-9b69e3ee460b","Type":"ContainerDied","Data":"96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284"} Nov 03 22:49:22.210979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.210928 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v" event={"ID":"03aadb43-1ccb-4185-af97-9b69e3ee460b","Type":"ContainerDied","Data":"eced1e2bf4d80056901bf1355aead38c8d2f7bc6906704e59f6526d41f7f3048"} Nov 03 22:49:22.210979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.210957 5272 scope.go:122] "RemoveContainer" containerID="96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284" Nov 03 22:49:22.222797 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.222767 5272 scope.go:122] "RemoveContainer" containerID="a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa" Nov 03 22:49:22.225209 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225183 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-certs\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225232 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-podinfo\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225321 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-credential-socket\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225448 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-certs" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:22.225537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225474 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-envoy\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225546 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-data\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225639 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03aadb43-1ccb-4185-af97-9b69e3ee460b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03aadb43-1ccb-4185-af97-9b69e3ee460b-istiod-ca-cert\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225668 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-kube-api-access-5dkmr\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-kube-api-access-5dkmr\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225699 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-socket\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225665 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-credential-socket" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:22.225907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225783 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-token\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-token\") pod \"03aadb43-1ccb-4185-af97-9b69e3ee460b\" (UID: \"03aadb43-1ccb-4185-af97-9b69e3ee460b\") " Nov 03 22:49:22.225907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.225767 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-data" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:22.226049 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.226011 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-socket" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:22.226434 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.226409 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03aadb43-1ccb-4185-af97-9b69e3ee460b-istiod-ca-cert" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:22.227874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.227245 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.227874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.227274 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.227874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.227288 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.227874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.227302 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03aadb43-1ccb-4185-af97-9b69e3ee460b-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.227874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.227316 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.228435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.228404 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-kube-api-access-5dkmr" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "kube-api-access-5dkmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:22.229219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.229187 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-podinfo" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:22.229318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.229217 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-token" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:22.229318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.229218 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-envoy" pod "03aadb43-1ccb-4185-af97-9b69e3ee460b" (UID: "03aadb43-1ccb-4185-af97-9b69e3ee460b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:22.234595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.234576 5272 scope.go:122] "RemoveContainer" containerID="96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284" Nov 03 22:49:22.234914 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:22.234891 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284\": container with ID starting with 96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284 not found: ID does not exist" containerID="96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284" Nov 03 22:49:22.234978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.234922 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284"} err="failed to get container status \"96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284\": rpc error: code = NotFound desc = could not find container \"96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284\": container with ID starting with 96b583a7c52182ae6af9f8338e0a6c0e86961b77a3a1f3ef281635eac1846284 not found: ID does not exist" Nov 03 22:49:22.234978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.234939 5272 scope.go:122] "RemoveContainer" containerID="a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa" Nov 03 22:49:22.235170 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:22.235155 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa\": container with ID starting with a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa not found: ID does not exist" containerID="a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa" Nov 03 22:49:22.235214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.235182 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa"} err="failed to get container status \"a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa\": rpc error: code = NotFound desc = could not find container \"a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa\": container with ID starting with a92802c18ccb3eafa5d192db954b378f50c78a55fe781514d00ed507471552aa not found: ID does not exist" Nov 03 22:49:22.328533 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.328498 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dkmr\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-kube-api-access-5dkmr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.328533 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.328531 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.328533 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.328541 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.328773 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.328550 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03aadb43-1ccb-4185-af97-9b69e3ee460b-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:22.515469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.515380 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv"] Nov 03 22:49:22.516552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.516521 5272 cpu_manager.go:414] "containerMap: removing container" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" Nov 03 22:49:22.516552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.516549 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" Nov 03 22:49:22.516748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.516569 5272 cpu_manager.go:414] "containerMap: removing container" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" Nov 03 22:49:22.516748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.516578 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" Nov 03 22:49:22.516912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.516795 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" Nov 03 22:49:22.522774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.522754 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.580083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.577013 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12450a06\"/\"openshift-service-ca.crt\"" Nov 03 22:49:22.580083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.577043 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12450a06\"/\"default-dockercfg-7k7zt\"" Nov 03 22:49:22.580083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.577103 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12450a06\"/\"e2e-lifecycle-model-12450a06-kserve-self-signed-certs\"" Nov 03 22:49:22.580083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.577163 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12450a06\"/\"kube-root-ca.crt\"" Nov 03 22:49:22.581386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.581355 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv"] Nov 03 22:49:22.633096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.633064 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqr5p\" (UniqueName: \"kubernetes.io/projected/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-kube-api-access-vqr5p\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.633282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.633183 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-dshm\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.633342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.633319 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tls-certs\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.633386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.633376 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-home\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.633424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.633395 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-model-cache\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.633472 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.633435 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tmp-dir\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.642136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.642108 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v"] Nov 03 22:49:22.679882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.679293 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e7a91ca4-openshift-default-59b8db5485-7fm8v"] Nov 03 22:49:22.689355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.688421 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7"] Nov 03 22:49:22.689355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.688909 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" containerID="cri-o://6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520" gracePeriod=30 Nov 03 22:49:22.695279 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.694067 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq"] Nov 03 22:49:22.695279 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.694581 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" containerID="cri-o://25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e" gracePeriod=30 Nov 03 22:49:22.702499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.700731 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6"] Nov 03 22:49:22.702499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.701059 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" containerID="cri-o://bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3" gracePeriod=30 Nov 03 22:49:22.704982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.704946 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf"] Nov 03 22:49:22.705217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.705191 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" containerID="cri-o://7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705" gracePeriod=30 Nov 03 22:49:22.735090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735059 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tls-certs\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735118 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-home\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735139 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-model-cache\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735171 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tmp-dir\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735306 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqr5p\" (UniqueName: \"kubernetes.io/projected/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-kube-api-access-vqr5p\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735393 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-dshm\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735575 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-home\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735638 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-model-cache\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.735683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.735651 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tmp-dir\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.737721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.737693 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-dshm\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.737935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.737916 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tls-certs\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.774057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.770028 5272 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:49:22.774057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.770088 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 22:49:22.794903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.794874 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqr5p\" (UniqueName: \"kubernetes.io/projected/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-kube-api-access-vqr5p\") pod \"e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:22.839789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:22.839751 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:23.329162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:23.329128 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv"] Nov 03 22:49:23.336047 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:23.336016 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0aa3a5e_e0f7_44b8_89f4_2556429e0dd2.slice/crio-747bda5ab502edfdf81a6a2a16e42a68fd996b4a05b72074ba883c9876043af1 WatchSource:0}: Error finding container 747bda5ab502edfdf81a6a2a16e42a68fd996b4a05b72074ba883c9876043af1: Status 404 returned error can't find the container with id 747bda5ab502edfdf81a6a2a16e42a68fd996b4a05b72074ba883c9876043af1 Nov 03 22:49:23.496480 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:23.496438 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" path="/var/lib/kubelet/pods/03aadb43-1ccb-4185-af97-9b69e3ee460b/volumes" Nov 03 22:49:23.767885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:23.763891 5272 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:49:23.767885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:23.763957 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 22:49:24.315986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:24.315948 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" event={"ID":"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2","Type":"ContainerStarted","Data":"40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db"} Nov 03 22:49:24.315986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:24.315993 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" event={"ID":"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2","Type":"ContainerStarted","Data":"747bda5ab502edfdf81a6a2a16e42a68fd996b4a05b72074ba883c9876043af1"} Nov 03 22:49:24.432300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:24.432236 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" podStartSLOduration=2.432219268 podStartE2EDuration="2.432219268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:24.430090985 +0000 UTC m=+1545.052399293" watchObservedRunningTime="2026-11-03 22:49:24.432219268 +0000 UTC m=+1545.054527575" Nov 03 22:49:24.502870 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:24.502834 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:49:24.768599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:24.764960 5272 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:49:24.768599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:24.765025 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 22:49:24.840594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:24.840554 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:25.068319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:25.068286 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:49:25.501314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:25.501238 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:49:25.501314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:25.501288 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:49:25.768358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:25.768250 5272 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:49:25.768358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:25.768326 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 22:49:26.371070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:26.371029 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:49:26.759700 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:26.759614 5272 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:49:26.759700 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:26.759675 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 22:49:26.909616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:26.909582 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:49:27.148960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:27.148929 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:49:27.764434 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:27.764402 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:49:27.765915 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:27.765893 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 22:49:27.773084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:27.773054 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:49:27.773198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:27.773107 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:49:27.787600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:27.787570 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:49:27.787722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:27.787626 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:49:28.069149 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.069078 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:49:28.072299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.072260 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:49:28.077422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.077393 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:49:28.247162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247123 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-credential-socket\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.247410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247194 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-data\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.247410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247279 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-envoy\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.247410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247310 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-data\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.247649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247404 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-istio-token\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-istio-token\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.247649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247488 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-podinfo\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.247649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247517 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a883e441-712e-4aa9-88a8-04e118a4b594-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a883e441-712e-4aa9-88a8-04e118a4b594-istio-podinfo\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.247649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247574 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-data\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.247649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247606 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-certs\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.248038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247692 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7faef726-470f-4c18-99cd-4569fdb2d878-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faef726-470f-4c18-99cd-4569fdb2d878-istiod-ca-cert\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.248038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247782 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-istio-token\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-istio-token\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.248038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247862 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-socket\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.248038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.247895 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-credential-socket\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.251137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.251087 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-certs\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.251540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.251472 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-envoy\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.251739 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-kube-api-access-8d66r\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-kube-api-access-8d66r\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.251824 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-kube-api-access-8zxgt\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-kube-api-access-8zxgt\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.251899 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-socket\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.251966 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istiod-ca-cert\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252007 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a883e441-712e-4aa9-88a8-04e118a4b594-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a883e441-712e-4aa9-88a8-04e118a4b594-istiod-ca-cert\") pod \"a883e441-712e-4aa9-88a8-04e118a4b594\" (UID: \"a883e441-712e-4aa9-88a8-04e118a4b594\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252069 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-credential-socket\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252134 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-certs\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252179 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7faef726-470f-4c18-99cd-4569fdb2d878-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faef726-470f-4c18-99cd-4569fdb2d878-istio-podinfo\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252236 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-kube-api-access-f6rh8\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-kube-api-access-f6rh8\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252316 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-token\") pod \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\" (UID: \"fb74ea1e-7dec-463e-9076-6fbedfb296ba\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252399 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-envoy\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.253653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.252445 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-socket\") pod \"7faef726-470f-4c18-99cd-4569fdb2d878\" (UID: \"7faef726-470f-4c18-99cd-4569fdb2d878\") " Nov 03 22:49:28.254707 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.254505 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-credential-socket" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.256213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.255351 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-data" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.257259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.256916 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-socket" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.257597 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.257419 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-credential-socket" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.257940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.257758 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-certs" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.258283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.258069 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-data" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.258814 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.258577 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istiod-ca-cert" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:28.258814 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.258602 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-socket" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.259007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.258872 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-socket" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.259853 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.259492 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7faef726-470f-4c18-99cd-4569fdb2d878-istiod-ca-cert" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:28.264566 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.261722 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-data" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.264566 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.262964 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-certs" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.264566 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.263212 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-credential-socket" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.264566 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.263394 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-certs" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.264566 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.263888 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a883e441-712e-4aa9-88a8-04e118a4b594-istiod-ca-cert" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:28.265276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.265234 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-envoy" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.266247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.266209 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-podinfo" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:28.266453 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.266420 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a883e441-712e-4aa9-88a8-04e118a4b594-istio-podinfo" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:28.268382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.268052 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7faef726-470f-4c18-99cd-4569fdb2d878-istio-podinfo" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:28.268382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.268161 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-envoy" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.268382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.268283 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-kube-api-access-8zxgt" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "kube-api-access-8zxgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.268382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.268356 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-istio-token" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.268743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.268713 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-envoy" pod "7faef726-470f-4c18-99cd-4569fdb2d878" (UID: "7faef726-470f-4c18-99cd-4569fdb2d878"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.269232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.269194 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-token" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.270675 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.270640 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-istio-token" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.272959 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.272930 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-kube-api-access-8d66r" pod "a883e441-712e-4aa9-88a8-04e118a4b594" (UID: "a883e441-712e-4aa9-88a8-04e118a4b594"). InnerVolumeSpecName "kube-api-access-8d66r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.275904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.274854 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-kube-api-access-f6rh8" pod "fb74ea1e-7dec-463e-9076-6fbedfb296ba" (UID: "fb74ea1e-7dec-463e-9076-6fbedfb296ba"). InnerVolumeSpecName "kube-api-access-f6rh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.355503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355460 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355503 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355502 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a883e441-712e-4aa9-88a8-04e118a4b594-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355517 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355529 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355542 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faef726-470f-4c18-99cd-4569fdb2d878-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355558 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rh8\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-kube-api-access-f6rh8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355571 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355582 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355596 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355608 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355620 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355631 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355643 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355655 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355668 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355682 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a883e441-712e-4aa9-88a8-04e118a4b594-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355695 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355708 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faef726-470f-4c18-99cd-4569fdb2d878-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.355725 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355721 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faef726-470f-4c18-99cd-4569fdb2d878-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355748 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355765 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355778 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355793 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355825 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea1e-7dec-463e-9076-6fbedfb296ba-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355841 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d66r\" (UniqueName: \"kubernetes.io/projected/a883e441-712e-4aa9-88a8-04e118a4b594-kube-api-access-8d66r\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355855 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zxgt\" (UniqueName: \"kubernetes.io/projected/7faef726-470f-4c18-99cd-4569fdb2d878-kube-api-access-8zxgt\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.356608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.355869 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a883e441-712e-4aa9-88a8-04e118a4b594-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.393353 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.393316 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:49:28.475054 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.475016 5272 generic.go:356] "Generic (PLEG): container finished" podID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerID="bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3" exitCode=0 Nov 03 22:49:28.475269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.475079 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" Nov 03 22:49:28.475269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.475103 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" event={"ID":"fb74ea1e-7dec-463e-9076-6fbedfb296ba","Type":"ContainerDied","Data":"bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3"} Nov 03 22:49:28.475398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.475268 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6" event={"ID":"fb74ea1e-7dec-463e-9076-6fbedfb296ba","Type":"ContainerDied","Data":"dde924c957ffc1feddf2b52caf7e3c643ddd62c535c94e04c21f63080dbf5706"} Nov 03 22:49:28.475398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.475322 5272 scope.go:122] "RemoveContainer" containerID="bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3" Nov 03 22:49:28.482523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.478465 5272 generic.go:356] "Generic (PLEG): container finished" podID="a883e441-712e-4aa9-88a8-04e118a4b594" containerID="25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e" exitCode=0 Nov 03 22:49:28.482523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.478762 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" event={"ID":"a883e441-712e-4aa9-88a8-04e118a4b594","Type":"ContainerDied","Data":"25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e"} Nov 03 22:49:28.482523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.478909 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" event={"ID":"a883e441-712e-4aa9-88a8-04e118a4b594","Type":"ContainerDied","Data":"6886d7623ad1120081e185f305e5fcfd82c0e10ae850a1bb7ee00c80577648c6"} Nov 03 22:49:28.482523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.479029 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq" Nov 03 22:49:28.488698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.487931 5272 generic.go:356] "Generic (PLEG): container finished" podID="7faef726-470f-4c18-99cd-4569fdb2d878" containerID="6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520" exitCode=0 Nov 03 22:49:28.488698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.487979 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" Nov 03 22:49:28.488698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.488015 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" event={"ID":"7faef726-470f-4c18-99cd-4569fdb2d878","Type":"ContainerDied","Data":"6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520"} Nov 03 22:49:28.488698 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.488082 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7" event={"ID":"7faef726-470f-4c18-99cd-4569fdb2d878","Type":"ContainerDied","Data":"3013663d91807d9ffe3f6bb56fe74932e9cff88e4b0222e00a07ea8e615547e6"} Nov 03 22:49:28.489919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.489897 5272 generic.go:356] "Generic (PLEG): container finished" podID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerID="7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705" exitCode=0 Nov 03 22:49:28.490046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.489944 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" Nov 03 22:49:28.490046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.489974 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" event={"ID":"cc5534eb-4e13-448a-a1a6-1628d4ff663c","Type":"ContainerDied","Data":"7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705"} Nov 03 22:49:28.490046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.490023 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf" event={"ID":"cc5534eb-4e13-448a-a1a6-1628d4ff663c","Type":"ContainerDied","Data":"58d063ba0b8c755c318544e617d98fab7326ce11e70ea4e7c603741e981b2be8"} Nov 03 22:49:28.491445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.491420 5272 scope.go:122] "RemoveContainer" containerID="bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3" Nov 03 22:49:28.491879 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:28.491853 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3\": container with ID starting with bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3 not found: ID does not exist" containerID="bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3" Nov 03 22:49:28.491965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.491887 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3"} err="failed to get container status \"bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3\": rpc error: code = NotFound desc = could not find container \"bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3\": container with ID starting with bd19720e13b96aab710b5d935d00bf56e46086d6217edb2791c654e5537468a3 not found: ID does not exist" Nov 03 22:49:28.491965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.491904 5272 scope.go:122] "RemoveContainer" containerID="25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e" Nov 03 22:49:28.503339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.503318 5272 scope.go:122] "RemoveContainer" containerID="25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e" Nov 03 22:49:28.503682 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:28.503655 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e\": container with ID starting with 25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e not found: ID does not exist" containerID="25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e" Nov 03 22:49:28.503779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.503696 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e"} err="failed to get container status \"25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e\": rpc error: code = NotFound desc = could not find container \"25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e\": container with ID starting with 25e74258d7047aff59062f8f5b76e33d47a761555727b41983792f37614bfe5e not found: ID does not exist" Nov 03 22:49:28.503779 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.503726 5272 scope.go:122] "RemoveContainer" containerID="6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520" Nov 03 22:49:28.517883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.517857 5272 scope.go:122] "RemoveContainer" containerID="6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520" Nov 03 22:49:28.518242 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:28.518218 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520\": container with ID starting with 6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520 not found: ID does not exist" containerID="6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520" Nov 03 22:49:28.518300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.518250 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520"} err="failed to get container status \"6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520\": rpc error: code = NotFound desc = could not find container \"6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520\": container with ID starting with 6a9892c6dfe810f3918dc540cdab1c1c72ec55e0d2973e2ff0ee9046b97cc520 not found: ID does not exist" Nov 03 22:49:28.518300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.518270 5272 scope.go:122] "RemoveContainer" containerID="7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705" Nov 03 22:49:28.529245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.529224 5272 scope.go:122] "RemoveContainer" containerID="7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705" Nov 03 22:49:28.529547 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:28.529530 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705\": container with ID starting with 7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705 not found: ID does not exist" containerID="7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705" Nov 03 22:49:28.529590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.529555 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705"} err="failed to get container status \"7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705\": rpc error: code = NotFound desc = could not find container \"7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705\": container with ID starting with 7d9bd796654c46e2855eb3069dcd6bf4523bcf6579d9005bebcd9975b5e0d705 not found: ID does not exist" Nov 03 22:49:28.558422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558387 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-token\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-token\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.558627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558459 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-kube-api-access-d2wpm\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.558627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558487 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-envoy\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.558627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558535 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-podinfo\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.558904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558630 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-data\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.558904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558734 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-socket\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.558904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558776 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-certs\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.558904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558846 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istiod-ca-cert\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.559188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.558969 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-credential-socket\") pod \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\" (UID: \"cc5534eb-4e13-448a-a1a6-1628d4ff663c\") " Nov 03 22:49:28.559188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.559049 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-data" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.559356 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.559214 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-socket" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.559356 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.559272 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-certs" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.559462 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.559369 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istiod-ca-cert" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:28.559462 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.559450 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-credential-socket" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.560028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.560002 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.560207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.560060 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.560207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.560099 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.560207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.560138 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.560207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.560178 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.560883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.560857 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-kube-api-access-d2wpm" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "kube-api-access-d2wpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.561213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.561189 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-token" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:28.561453 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.561430 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-podinfo" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:28.562542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.562520 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-envoy" pod "cc5534eb-4e13-448a-a1a6-1628d4ff663c" (UID: "cc5534eb-4e13-448a-a1a6-1628d4ff663c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:28.667001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.666965 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.667001 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.667000 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wpm\" (UniqueName: \"kubernetes.io/projected/cc5534eb-4e13-448a-a1a6-1628d4ff663c-kube-api-access-d2wpm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.667222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.667015 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.667222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.667030 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5534eb-4e13-448a-a1a6-1628d4ff663c-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:49:28.714395 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.714347 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq"] Nov 03 22:49:28.718354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.718326 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qkcnq"] Nov 03 22:49:28.821123 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.821088 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6"] Nov 03 22:49:28.825157 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.825127 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-k46v6"] Nov 03 22:49:28.927107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.927069 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7"] Nov 03 22:49:28.932364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:28.932285 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-rj8x7"] Nov 03 22:49:29.034840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.034777 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf"] Nov 03 22:49:29.040816 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.040773 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-qnfhf"] Nov 03 22:49:29.582827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.580127 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" path="/var/lib/kubelet/pods/7faef726-470f-4c18-99cd-4569fdb2d878/volumes" Nov 03 22:49:29.582827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.580956 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" path="/var/lib/kubelet/pods/a883e441-712e-4aa9-88a8-04e118a4b594/volumes" Nov 03 22:49:29.582827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.581575 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" path="/var/lib/kubelet/pods/cc5534eb-4e13-448a-a1a6-1628d4ff663c/volumes" Nov 03 22:49:29.582827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.582756 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" path="/var/lib/kubelet/pods/fb74ea1e-7dec-463e-9076-6fbedfb296ba/volumes" Nov 03 22:49:29.589093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.589055 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd"] Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591201 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591227 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591265 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591273 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591289 5272 cpu_manager.go:414] "containerMap: removing container" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591296 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591318 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591326 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591461 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb74ea1e-7dec-463e-9076-6fbedfb296ba" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591472 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5534eb-4e13-448a-a1a6-1628d4ff663c" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591483 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a883e441-712e-4aa9-88a8-04e118a4b594" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591501 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7faef726-470f-4c18-99cd-4569fdb2d878" containerName="istio-proxy" Nov 03 22:49:29.592834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.591511 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aadb43-1ccb-4185-af97-9b69e3ee460b" containerName="istio-proxy" Nov 03 22:49:29.775824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.769564 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd"] Nov 03 22:49:29.775824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.769620 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9"] Nov 03 22:49:29.775824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.769834 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:29.790825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.787674 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:29.951895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.951048 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9"] Nov 03 22:49:29.969186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.969152 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-6a3169-dockercfg-4z76b\"" Nov 03 22:49:29.970074 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:29.970050 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-6a3169\"" Nov 03 22:49:30.075751 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.075717 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:30.076046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.076026 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8bn\" (UniqueName: \"kubernetes.io/projected/9180335f-26ce-4ff6-b046-a92e3f42303d-kube-api-access-gt8bn\") pod \"payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:30.076215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.076185 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9180335f-26ce-4ff6-b046-a92e3f42303d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:30.076331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.076311 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8qm\" (UniqueName: \"kubernetes.io/projected/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-kube-api-access-tm8qm\") pod \"payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:30.177513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.177466 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:30.177726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.177636 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8bn\" (UniqueName: \"kubernetes.io/projected/9180335f-26ce-4ff6-b046-a92e3f42303d-kube-api-access-gt8bn\") pod \"payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:30.177726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.177703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9180335f-26ce-4ff6-b046-a92e3f42303d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:30.177904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.177744 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8qm\" (UniqueName: \"kubernetes.io/projected/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-kube-api-access-tm8qm\") pod \"payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:30.178379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.178351 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:30.178492 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.178469 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9180335f-26ce-4ff6-b046-a92e3f42303d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:30.235464 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.235389 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8qm\" (UniqueName: \"kubernetes.io/projected/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-kube-api-access-tm8qm\") pod \"payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:30.235603 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.235501 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8bn\" (UniqueName: \"kubernetes.io/projected/9180335f-26ce-4ff6-b046-a92e3f42303d-kube-api-access-gt8bn\") pod \"payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:30.295864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.295787 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:30.309733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.309703 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:30.684648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.684611 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd"] Nov 03 22:49:30.730448 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:30.730417 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8b506e_c569_401d_a1e3_b2bc7ec06db2.slice/crio-f659a71f2ed0a9366a80f066b7757cb908bf89a005022ca3c0f445a8c57db8d3 WatchSource:0}: Error finding container f659a71f2ed0a9366a80f066b7757cb908bf89a005022ca3c0f445a8c57db8d3: Status 404 returned error can't find the container with id f659a71f2ed0a9366a80f066b7757cb908bf89a005022ca3c0f445a8c57db8d3 Nov 03 22:49:30.737970 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:30.737786 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9"] Nov 03 22:49:30.745310 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:30.745284 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9180335f_26ce_4ff6_b046_a92e3f42303d.slice/crio-38f9683b6925b02f1832c8fb124029d0c5cb4a5b4aa75bee96c9b476aa2a4a8f WatchSource:0}: Error finding container 38f9683b6925b02f1832c8fb124029d0c5cb4a5b4aa75bee96c9b476aa2a4a8f: Status 404 returned error can't find the container with id 38f9683b6925b02f1832c8fb124029d0c5cb4a5b4aa75bee96c9b476aa2a4a8f Nov 03 22:49:31.648471 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.648429 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" event={"ID":"9180335f-26ce-4ff6-b046-a92e3f42303d","Type":"ContainerStarted","Data":"ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73"} Nov 03 22:49:31.648983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.648500 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:31.648983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.648525 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" event={"ID":"9180335f-26ce-4ff6-b046-a92e3f42303d","Type":"ContainerStarted","Data":"38f9683b6925b02f1832c8fb124029d0c5cb4a5b4aa75bee96c9b476aa2a4a8f"} Nov 03 22:49:31.650333 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.650306 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" event={"ID":"1f8b506e-c569-401d-a1e3-b2bc7ec06db2","Type":"ContainerStarted","Data":"665982b76d245edaa3eba3a55cbff62611f483ba54904b6f08448fced3b8823b"} Nov 03 22:49:31.650435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.650359 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" event={"ID":"1f8b506e-c569-401d-a1e3-b2bc7ec06db2","Type":"ContainerStarted","Data":"f659a71f2ed0a9366a80f066b7757cb908bf89a005022ca3c0f445a8c57db8d3"} Nov 03 22:49:31.650435 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.650401 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:31.760976 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.760909 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" podStartSLOduration=2.760891987 podStartE2EDuration="2.760891987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:31.756626811 +0000 UTC m=+1552.378935119" watchObservedRunningTime="2026-11-03 22:49:31.760891987 +0000 UTC m=+1552.383200295" Nov 03 22:49:31.867030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:31.866971 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" podStartSLOduration=2.866955115 podStartE2EDuration="2.866955115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:31.864948522 +0000 UTC m=+1552.487256830" watchObservedRunningTime="2026-11-03 22:49:31.866955115 +0000 UTC m=+1552.489263424" Nov 03 22:49:32.841015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:32.840979 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:32.854125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:32.854095 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:33.680222 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:33.680189 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:49:41.890434 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.890392 5272 scope.go:122] "RemoveContainer" containerID="1163c9eedd7b41a426cb0c9b785e6074857450c821b293a00f51a3c1c9efb56e" Nov 03 22:49:41.901548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.901523 5272 scope.go:122] "RemoveContainer" containerID="c42add853035352c3270e1544ce5fc3aa26b1fb5801b53ac7278ca35f1bf94e0" Nov 03 22:49:41.914553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.914530 5272 scope.go:122] "RemoveContainer" containerID="09e36753d2a7a9d2329f4e95ece33e7009b4747fb42ae5c27fde56e38ca291d1" Nov 03 22:49:41.924960 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.924935 5272 scope.go:122] "RemoveContainer" containerID="bf6b0c2e3aad7b0eb3309795d94cfb3a269277d56e0fd450e9ed8813ca63da0b" Nov 03 22:49:41.936474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.936457 5272 scope.go:122] "RemoveContainer" containerID="8999cffc30d2ed35822065bca1ab7a7a9faed9bb933f3037d2bb59f050a86066" Nov 03 22:49:41.947109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.947089 5272 scope.go:122] "RemoveContainer" containerID="fb3b960c60460c3c5b5eea654f3a51dbd3e2f03c9a1b5a3a0e4dec948a6927d3" Nov 03 22:49:41.957987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.957970 5272 scope.go:122] "RemoveContainer" containerID="257c29a73804e02b30efa9e7862c6203f23eafe084fbbd96a5d9f55aa308445e" Nov 03 22:49:41.969414 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.969390 5272 scope.go:122] "RemoveContainer" containerID="b6808fe39ecc8074dc0519866df7bf9d5542338c21505078a66aa33fe7aa5f90" Nov 03 22:49:41.980106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.980080 5272 scope.go:122] "RemoveContainer" containerID="440bacf7145a44c2a66c7debf193fb61186ba4622d930151bf1d48d18c5f6f53" Nov 03 22:49:41.989752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.989728 5272 scope.go:122] "RemoveContainer" containerID="23819f23ec4170e098e0f0394c5d672987ced3ccb2badf5fa17e8ff177113459" Nov 03 22:49:41.999076 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:41.999054 5272 scope.go:122] "RemoveContainer" containerID="4c99c0692b3ffe5753cfc8e51b4e399f6dcdad106d1d8ee80375acbec168b980" Nov 03 22:49:42.008551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.008533 5272 scope.go:122] "RemoveContainer" containerID="753d6fd59dca02a82aa8b1f3e8323603ddbbd611609c4eac8282a8bae7514685" Nov 03 22:49:42.022899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.022878 5272 scope.go:122] "RemoveContainer" containerID="3d1b8a4420d3d86bb7a1252f000a135a34f078c88b5250bdf1bbc7da8a32ee79" Nov 03 22:49:42.032872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.032853 5272 scope.go:122] "RemoveContainer" containerID="c6fc10f0875cd9125843e54a2b0c77d899b63a61def4238a59a16dfa15ebee36" Nov 03 22:49:42.805825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.776243 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh"] Nov 03 22:49:42.850116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.848871 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh"] Nov 03 22:49:42.850116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.848969 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:49:42.850116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.849020 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:49:42.850116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.850020 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.937398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937363 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-credential-socket\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.937885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937438 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-data\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.937885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937535 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-certs\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.937885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937593 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-istio-token\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.937885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937752 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-envoy\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.937885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937774 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45585d1a-823c-4006-bb93-f257c3dcb895-istio-podinfo\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.937885 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937831 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-socket\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.938157 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.937957 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhbk\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-kube-api-access-zdhbk\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.938157 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.938065 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45585d1a-823c-4006-bb93-f257c3dcb895-istiod-ca-cert\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:42.946129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:42.946098 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0c21c7-openshift-default-dockercfg-zxsds\"" Nov 03 22:49:43.039869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.039793 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-credential-socket\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.039912 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-data\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.039952 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-certs\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.039983 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-istio-token\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040004 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-envoy\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040019 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45585d1a-823c-4006-bb93-f257c3dcb895-istio-podinfo\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040049 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-socket\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040092 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhbk\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-kube-api-access-zdhbk\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040155 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45585d1a-823c-4006-bb93-f257c3dcb895-istiod-ca-cert\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040281 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040246 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-credential-socket\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040515 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040299 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-certs\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040515 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040355 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-data\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040547 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-socket\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.040973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.040949 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45585d1a-823c-4006-bb93-f257c3dcb895-istiod-ca-cert\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.042534 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.042508 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-envoy\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.042838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.042781 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45585d1a-823c-4006-bb93-f257c3dcb895-istio-podinfo\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.116876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.113297 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhbk\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-kube-api-access-zdhbk\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.135871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.135545 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-istio-token\") pod \"e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.198838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.195992 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:43.657921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.657886 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh"] Nov 03 22:49:43.662817 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:43.662773 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45585d1a_823c_4006_bb93_f257c3dcb895.slice/crio-201ffe147f82fbffd5763a8bf776035e0f6f170b0cdb17cc2527b0f56c579cd2 WatchSource:0}: Error finding container 201ffe147f82fbffd5763a8bf776035e0f6f170b0cdb17cc2527b0f56c579cd2: Status 404 returned error can't find the container with id 201ffe147f82fbffd5763a8bf776035e0f6f170b0cdb17cc2527b0f56c579cd2 Nov 03 22:49:43.665613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.665581 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:43.665693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.665649 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:43.665693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.665680 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:49:43.827499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.827462 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" event={"ID":"45585d1a-823c-4006-bb93-f257c3dcb895","Type":"ContainerStarted","Data":"9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf"} Nov 03 22:49:43.827668 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.827521 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" event={"ID":"45585d1a-823c-4006-bb93-f257c3dcb895","Type":"ContainerStarted","Data":"201ffe147f82fbffd5763a8bf776035e0f6f170b0cdb17cc2527b0f56c579cd2"} Nov 03 22:49:43.945199 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:43.945090 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podStartSLOduration=1.945071396 podStartE2EDuration="1.945071396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:43.94130949 +0000 UTC m=+1564.563617799" watchObservedRunningTime="2026-11-03 22:49:43.945071396 +0000 UTC m=+1564.567379709" Nov 03 22:49:44.196420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:44.196329 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:44.202986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:44.202958 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:44.843614 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:44.843580 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:44.844610 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:44.844586 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:49:51.823999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:51.823955 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv"] Nov 03 22:49:51.830211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:51.830184 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:51.911331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:51.911293 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv"] Nov 03 22:49:51.932417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:51.921047 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:51.932417 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:51.921274 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7l4s\" (UniqueName: \"kubernetes.io/projected/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-kube-api-access-x7l4s\") pod \"payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:51.949876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:51.949097 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0c21c7\"" Nov 03 22:49:51.949876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:51.949240 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0c21c7-dockercfg-bqkmn\"" Nov 03 22:49:52.023776 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.023737 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:52.024195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.024166 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7l4s\" (UniqueName: \"kubernetes.io/projected/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-kube-api-access-x7l4s\") pod \"payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:52.071938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.071900 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:52.120933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.112730 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7l4s\" (UniqueName: \"kubernetes.io/projected/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-kube-api-access-x7l4s\") pod \"payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:52.154927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.153793 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:52.157271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.157237 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr"] Nov 03 22:49:52.167791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.167054 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr"] Nov 03 22:49:52.167791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.167202 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.351354 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.351317 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/8c66265a-7fb0-4516-b53c-ecf85bb84806-kube-api-access-hznjh\") pod \"payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.351534 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.351392 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c66265a-7fb0-4516-b53c-ecf85bb84806-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.452630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.452546 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/8c66265a-7fb0-4516-b53c-ecf85bb84806-kube-api-access-hznjh\") pod \"payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.452630 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.452597 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c66265a-7fb0-4516-b53c-ecf85bb84806-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.453391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.453362 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c66265a-7fb0-4516-b53c-ecf85bb84806-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.509922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.509892 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/8c66265a-7fb0-4516-b53c-ecf85bb84806-kube-api-access-hznjh\") pod \"payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.548636 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.548585 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:52.582130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.582101 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv"] Nov 03 22:49:52.586014 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:52.585981 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0de1f4_768a_4c03_b2fa_db92db8d8b69.slice/crio-3fb596510b58f39655df8c6ad7e8f885cf2ac6fd86e20f19615b123f6ef4e065 WatchSource:0}: Error finding container 3fb596510b58f39655df8c6ad7e8f885cf2ac6fd86e20f19615b123f6ef4e065: Status 404 returned error can't find the container with id 3fb596510b58f39655df8c6ad7e8f885cf2ac6fd86e20f19615b123f6ef4e065 Nov 03 22:49:52.932610 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:52.932572 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr"] Nov 03 22:49:52.937845 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:49:52.937819 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c66265a_7fb0_4516_b53c_ecf85bb84806.slice/crio-e7e59fcb4c3a6a2b0c21513a559cef6bcb54709d653f8056bd7a02b73eff1335 WatchSource:0}: Error finding container e7e59fcb4c3a6a2b0c21513a559cef6bcb54709d653f8056bd7a02b73eff1335: Status 404 returned error can't find the container with id e7e59fcb4c3a6a2b0c21513a559cef6bcb54709d653f8056bd7a02b73eff1335 Nov 03 22:49:53.124857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.120778 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" event={"ID":"cc0de1f4-768a-4c03-b2fa-db92db8d8b69","Type":"ContainerStarted","Data":"d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d"} Nov 03 22:49:53.124857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.120876 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" event={"ID":"cc0de1f4-768a-4c03-b2fa-db92db8d8b69","Type":"ContainerStarted","Data":"3fb596510b58f39655df8c6ad7e8f885cf2ac6fd86e20f19615b123f6ef4e065"} Nov 03 22:49:53.124857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.120952 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:49:53.124857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.123294 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" event={"ID":"8c66265a-7fb0-4516-b53c-ecf85bb84806","Type":"ContainerStarted","Data":"adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2"} Nov 03 22:49:53.124857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.123341 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" event={"ID":"8c66265a-7fb0-4516-b53c-ecf85bb84806","Type":"ContainerStarted","Data":"e7e59fcb4c3a6a2b0c21513a559cef6bcb54709d653f8056bd7a02b73eff1335"} Nov 03 22:49:53.124857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.123371 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:49:53.231715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.231624 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" podStartSLOduration=2.231609578 podStartE2EDuration="2.231609578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:53.22920097 +0000 UTC m=+1573.851509277" watchObservedRunningTime="2026-11-03 22:49:53.231609578 +0000 UTC m=+1573.853917887" Nov 03 22:49:53.344147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:49:53.344078 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" podStartSLOduration=2.344018227 podStartE2EDuration="2.344018227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:53.337235189 +0000 UTC m=+1573.959543497" watchObservedRunningTime="2026-11-03 22:49:53.344018227 +0000 UTC m=+1573.966326539" Nov 03 22:49:54.502735 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:49:54.502704 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:50:02.277636 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.277601 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj"] Nov 03 22:50:02.283246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.283228 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.334298 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.334268 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj"] Nov 03 22:50:02.375974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.375927 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:50:02.375974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.375927 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:50:02.375974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.375960 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:50:02.378817 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.378779 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:50:02.402130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.402104 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.402278 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.402254 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtr6\" (UniqueName: \"kubernetes.io/projected/fe71c086-eea6-4678-a26f-95426eead8cc-kube-api-access-fxtr6\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.402398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.402383 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.504258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.504220 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.504442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.504367 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.504492 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.504467 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtr6\" (UniqueName: \"kubernetes.io/projected/fe71c086-eea6-4678-a26f-95426eead8cc-kube-api-access-fxtr6\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.505082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.505058 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.506662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.506641 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.562111 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.562080 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtr6\" (UniqueName: \"kubernetes.io/projected/fe71c086-eea6-4678-a26f-95426eead8cc-kube-api-access-fxtr6\") pod \"maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.598750 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.598714 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:02.980837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:02.980733 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj"] Nov 03 22:50:02.985607 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:50:02.985573 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe71c086_eea6_4678_a26f_95426eead8cc.slice/crio-3c6f5b2f5f7b04c62f0d88ec5fd9a16ae2ec8d5ee0aca6f2709eac20b18b7a0e WatchSource:0}: Error finding container 3c6f5b2f5f7b04c62f0d88ec5fd9a16ae2ec8d5ee0aca6f2709eac20b18b7a0e: Status 404 returned error can't find the container with id 3c6f5b2f5f7b04c62f0d88ec5fd9a16ae2ec8d5ee0aca6f2709eac20b18b7a0e Nov 03 22:50:03.205892 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:03.205854 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" event={"ID":"fe71c086-eea6-4678-a26f-95426eead8cc","Type":"ContainerStarted","Data":"3c6f5b2f5f7b04c62f0d88ec5fd9a16ae2ec8d5ee0aca6f2709eac20b18b7a0e"} Nov 03 22:50:04.134986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:04.134898 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:50:04.134986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:04.134979 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:50:04.215372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:04.215338 5272 generic.go:356] "Generic (PLEG): container finished" podID="fe71c086-eea6-4678-a26f-95426eead8cc" containerID="6b465e159ca0ff7ff0715d1a000cee9b7cd49f5e33a547eee2061b1b313ea0a0" exitCode=0 Nov 03 22:50:04.215546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:04.215380 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" event={"ID":"fe71c086-eea6-4678-a26f-95426eead8cc","Type":"ContainerDied","Data":"6b465e159ca0ff7ff0715d1a000cee9b7cd49f5e33a547eee2061b1b313ea0a0"} Nov 03 22:50:05.391343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.391313 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:05.550112 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.550076 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-xks-ca\") pod \"fe71c086-eea6-4678-a26f-95426eead8cc\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " Nov 03 22:50:05.550336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.550218 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe71c086-eea6-4678-a26f-95426eead8cc-kube-api-access-fxtr6\" (UniqueName: \"kubernetes.io/projected/fe71c086-eea6-4678-a26f-95426eead8cc-kube-api-access-fxtr6\") pod \"fe71c086-eea6-4678-a26f-95426eead8cc\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " Nov 03 22:50:05.550490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.550470 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-openshift-ca\") pod \"fe71c086-eea6-4678-a26f-95426eead8cc\" (UID: \"fe71c086-eea6-4678-a26f-95426eead8cc\") " Nov 03 22:50:05.550951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.550920 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-openshift-ca" pod "fe71c086-eea6-4678-a26f-95426eead8cc" (UID: "fe71c086-eea6-4678-a26f-95426eead8cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:05.551181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.551162 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:05.552599 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.552563 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-xks-ca" pod "fe71c086-eea6-4678-a26f-95426eead8cc" (UID: "fe71c086-eea6-4678-a26f-95426eead8cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:05.553867 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.553839 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe71c086-eea6-4678-a26f-95426eead8cc-kube-api-access-fxtr6" pod "fe71c086-eea6-4678-a26f-95426eead8cc" (UID: "fe71c086-eea6-4678-a26f-95426eead8cc"). InnerVolumeSpecName "kube-api-access-fxtr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:05.652730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.652694 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71c086-eea6-4678-a26f-95426eead8cc-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:05.652730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:05.652726 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxtr6\" (UniqueName: \"kubernetes.io/projected/fe71c086-eea6-4678-a26f-95426eead8cc-kube-api-access-fxtr6\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:06.233496 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:06.233464 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" Nov 03 22:50:06.233670 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:06.233498 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj" event={"ID":"fe71c086-eea6-4678-a26f-95426eead8cc","Type":"ContainerDied","Data":"3c6f5b2f5f7b04c62f0d88ec5fd9a16ae2ec8d5ee0aca6f2709eac20b18b7a0e"} Nov 03 22:50:06.233670 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:06.233551 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c6f5b2f5f7b04c62f0d88ec5fd9a16ae2ec8d5ee0aca6f2709eac20b18b7a0e" Nov 03 22:50:06.724131 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:06.724093 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:50:14.696251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:14.696211 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94"] Nov 03 22:50:14.798334 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:14.798298 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd"] Nov 03 22:50:19.697352 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:19.697307 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" podUID="453aaffc-2a66-4e0a-8371-64755cd3c0d5" containerName="payload-processing" containerID="cri-o://3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3" gracePeriod=25 Nov 03 22:50:19.799228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:19.799183 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" podUID="2a3d556b-898c-4567-96c0-a4278f64f24a" containerName="payload-pre-processing" containerID="cri-o://f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f" gracePeriod=25 Nov 03 22:50:20.027319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.027289 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:50:20.089121 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.089086 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:50:20.115593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.115561 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453aaffc-2a66-4e0a-8371-64755cd3c0d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453aaffc-2a66-4e0a-8371-64755cd3c0d5-plugins-config-volume\") pod \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " Nov 03 22:50:20.115966 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.115934 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453aaffc-2a66-4e0a-8371-64755cd3c0d5-plugins-config-volume" pod "453aaffc-2a66-4e0a-8371-64755cd3c0d5" (UID: "453aaffc-2a66-4e0a-8371-64755cd3c0d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:20.116047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.115973 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453aaffc-2a66-4e0a-8371-64755cd3c0d5-kube-api-access-7dvvw\" (UniqueName: \"kubernetes.io/projected/453aaffc-2a66-4e0a-8371-64755cd3c0d5-kube-api-access-7dvvw\") pod \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\" (UID: \"453aaffc-2a66-4e0a-8371-64755cd3c0d5\") " Nov 03 22:50:20.116505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.116488 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453aaffc-2a66-4e0a-8371-64755cd3c0d5-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:20.117989 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.117966 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453aaffc-2a66-4e0a-8371-64755cd3c0d5-kube-api-access-7dvvw" pod "453aaffc-2a66-4e0a-8371-64755cd3c0d5" (UID: "453aaffc-2a66-4e0a-8371-64755cd3c0d5"). InnerVolumeSpecName "kube-api-access-7dvvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:20.217365 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.217334 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a3d556b-898c-4567-96c0-a4278f64f24a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3d556b-898c-4567-96c0-a4278f64f24a-plugins-config-volume\") pod \"2a3d556b-898c-4567-96c0-a4278f64f24a\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " Nov 03 22:50:20.217532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.217421 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3d556b-898c-4567-96c0-a4278f64f24a-kube-api-access-g6pq8\" (UniqueName: \"kubernetes.io/projected/2a3d556b-898c-4567-96c0-a4278f64f24a-kube-api-access-g6pq8\") pod \"2a3d556b-898c-4567-96c0-a4278f64f24a\" (UID: \"2a3d556b-898c-4567-96c0-a4278f64f24a\") " Nov 03 22:50:20.217821 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.217774 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3d556b-898c-4567-96c0-a4278f64f24a-plugins-config-volume" pod "2a3d556b-898c-4567-96c0-a4278f64f24a" (UID: "2a3d556b-898c-4567-96c0-a4278f64f24a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:20.218080 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.218058 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvvw\" (UniqueName: \"kubernetes.io/projected/453aaffc-2a66-4e0a-8371-64755cd3c0d5-kube-api-access-7dvvw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:20.218142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.218083 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3d556b-898c-4567-96c0-a4278f64f24a-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:20.219531 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.219503 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3d556b-898c-4567-96c0-a4278f64f24a-kube-api-access-g6pq8" pod "2a3d556b-898c-4567-96c0-a4278f64f24a" (UID: "2a3d556b-898c-4567-96c0-a4278f64f24a"). InnerVolumeSpecName "kube-api-access-g6pq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:20.319663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.319625 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6pq8\" (UniqueName: \"kubernetes.io/projected/2a3d556b-898c-4567-96c0-a4278f64f24a-kube-api-access-g6pq8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:20.434381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.434340 5272 generic.go:356] "Generic (PLEG): container finished" podID="2a3d556b-898c-4567-96c0-a4278f64f24a" containerID="f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f" exitCode=0 Nov 03 22:50:20.434565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.434392 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" Nov 03 22:50:20.434565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.434427 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" event={"ID":"2a3d556b-898c-4567-96c0-a4278f64f24a","Type":"ContainerDied","Data":"f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f"} Nov 03 22:50:20.434565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.434483 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd" event={"ID":"2a3d556b-898c-4567-96c0-a4278f64f24a","Type":"ContainerDied","Data":"b890d4d8694cad8cdb59664e0bfc2858e14f4d8d866adf94a71219cdc48cf92c"} Nov 03 22:50:20.434565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.434507 5272 scope.go:122] "RemoveContainer" containerID="f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f" Nov 03 22:50:20.435995 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.435971 5272 generic.go:356] "Generic (PLEG): container finished" podID="453aaffc-2a66-4e0a-8371-64755cd3c0d5" containerID="3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3" exitCode=0 Nov 03 22:50:20.436103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.436012 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" Nov 03 22:50:20.436103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.436055 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" event={"ID":"453aaffc-2a66-4e0a-8371-64755cd3c0d5","Type":"ContainerDied","Data":"3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3"} Nov 03 22:50:20.436203 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.436121 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94" event={"ID":"453aaffc-2a66-4e0a-8371-64755cd3c0d5","Type":"ContainerDied","Data":"95460ee10576344f3a9f998195e6e8e35df46daa14127fffe2d4e69854cd8264"} Nov 03 22:50:20.447995 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.447978 5272 scope.go:122] "RemoveContainer" containerID="f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f" Nov 03 22:50:20.448276 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:50:20.448248 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f\": container with ID starting with f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f not found: ID does not exist" containerID="f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f" Nov 03 22:50:20.448347 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.448281 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f"} err="failed to get container status \"f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f\": rpc error: code = NotFound desc = could not find container \"f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f\": container with ID starting with f4b5e119f90e2375de267515d830e3ed2d1afa130d66967c487d03a61bb3a12f not found: ID does not exist" Nov 03 22:50:20.448347 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.448302 5272 scope.go:122] "RemoveContainer" containerID="3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3" Nov 03 22:50:20.457948 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.457929 5272 scope.go:122] "RemoveContainer" containerID="3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3" Nov 03 22:50:20.458196 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:50:20.458175 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3\": container with ID starting with 3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3 not found: ID does not exist" containerID="3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3" Nov 03 22:50:20.458252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.458204 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3"} err="failed to get container status \"3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3\": rpc error: code = NotFound desc = could not find container \"3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3\": container with ID starting with 3bcc091853454f5873eaa596d4c75543bb4d7346fe2bd3a7a03d602ec5e52bc3 not found: ID does not exist" Nov 03 22:50:20.591556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.591460 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd"] Nov 03 22:50:20.595456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.595430 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d18a5b05-6bc8f78cd7-xbnbd"] Nov 03 22:50:20.697782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.697749 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94"] Nov 03 22:50:20.704031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:20.704000 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d18a5b05-95bc9bb4-9tn94"] Nov 03 22:50:21.495943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:21.495911 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3d556b-898c-4567-96c0-a4278f64f24a" path="/var/lib/kubelet/pods/2a3d556b-898c-4567-96c0-a4278f64f24a/volumes" Nov 03 22:50:21.496466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:21.496450 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453aaffc-2a66-4e0a-8371-64755cd3c0d5" path="/var/lib/kubelet/pods/453aaffc-2a66-4e0a-8371-64755cd3c0d5/volumes" Nov 03 22:50:24.501986 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:50:24.501954 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:50:34.329921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:34.329887 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj"] Nov 03 22:50:34.376688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:34.376646 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d18a5b05-v6hqj"] Nov 03 22:50:35.495774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:35.495740 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe71c086-eea6-4678-a26f-95426eead8cc" path="/var/lib/kubelet/pods/fe71c086-eea6-4678-a26f-95426eead8cc/volumes" Nov 03 22:50:37.173326 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:37.173292 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck"] Nov 03 22:50:37.174185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:37.174142 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" containerID="cri-o://3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8" gracePeriod=30 Nov 03 22:50:37.199654 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:37.199585 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9"] Nov 03 22:50:37.199999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:37.199953 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" containerID="cri-o://28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f" gracePeriod=30 Nov 03 22:50:37.204902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:37.204873 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr"] Nov 03 22:50:37.205148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:37.205113 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" containerID="cri-o://eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b" gracePeriod=30 Nov 03 22:50:42.149465 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.149424 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:42.151869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.149500 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:42.347955 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.347919 5272 scope.go:122] "RemoveContainer" containerID="c315def20b5e272ce495b4c2c138e24b36db56083e8c28afb2b3069a7e92a7f9" Nov 03 22:50:42.404083 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.399339 5272 scope.go:122] "RemoveContainer" containerID="7208a7d0bf36728c5e93c2d3ce4a4167a4be39a1803d2cab5d4f31f29c202905" Nov 03 22:50:42.417301 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.417084 5272 scope.go:122] "RemoveContainer" containerID="f8c515ddcfb81c20fb025728f88fac8076e2bc1d89b92d73f88744d3d43510d3" Nov 03 22:50:42.431216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.431191 5272 scope.go:122] "RemoveContainer" containerID="331d9545a943efb32027dec03eb18315c7c8f0a7636431b372dcc54182c3fe37" Nov 03 22:50:42.447068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.446929 5272 scope.go:122] "RemoveContainer" containerID="f6dc0c4eae4ff606d2cfdd7726d5e6929cb8d37d5b84fe13ba7777f748af4676" Nov 03 22:50:42.519577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.519551 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:50:42.542841 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.539654 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:50:42.599925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.599901 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703407 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-socket\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703477 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-certs\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703510 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-data\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703568 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-envoy\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703601 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-podinfo\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703670 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-kube-api-access-8cgnd\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-kube-api-access-8cgnd\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703726 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03dd63fa-31cb-488e-8232-b07a170bfc44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dd63fa-31cb-488e-8232-b07a170bfc44-istiod-ca-cert\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703765 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-data\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703840 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-podinfo\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703902 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-certs\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.703964 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-data\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704011 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-credential-socket\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704078 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istiod-ca-cert\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.704110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704116 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-envoy\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704163 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-token\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-token\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704190 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-socket\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704253 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istiod-ca-cert\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704292 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-socket\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704349 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-envoy\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704378 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-credential-socket\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704426 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-podinfo\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704463 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-kube-api-access-97bws\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-kube-api-access-97bws\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704510 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-token\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-token\") pod \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\" (UID: \"e01a4b7c-5d26-4dd8-847d-17b838f3c07d\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704569 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-kube-api-access-25qcs\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-kube-api-access-25qcs\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704622 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-certs\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704653 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-credential-socket\") pod \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\" (UID: \"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3\") " Nov 03 22:50:42.705081 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.704707 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-token\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-token\") pod \"03dd63fa-31cb-488e-8232-b07a170bfc44\" (UID: \"03dd63fa-31cb-488e-8232-b07a170bfc44\") " Nov 03 22:50:42.706173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.706137 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-socket" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.706173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.706148 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-credential-socket" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.706729 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.706436 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istiod-ca-cert" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:42.706906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.706883 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-socket" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.707013 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.706913 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03dd63fa-31cb-488e-8232-b07a170bfc44-istiod-ca-cert" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:42.707186 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.707160 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-certs" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.707387 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.707364 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-data" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.708687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.707915 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-socket" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.708687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.708231 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istiod-ca-cert" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:42.708687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.708609 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-certs" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.709095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.708993 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-data" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.709502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.709232 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-credential-socket" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.709502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.709252 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-certs" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.709502 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.709473 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-credential-socket" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.709836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.709767 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-data" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.710651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.710621 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-token" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:42.712163 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.712136 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-kube-api-access-97bws" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "kube-api-access-97bws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:42.712614 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.712578 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-kube-api-access-8cgnd" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "kube-api-access-8cgnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:42.712747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.712725 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-podinfo" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:42.713175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.713151 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-envoy" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.713285 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.713217 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-token" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:42.713351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.713300 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-envoy" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.713442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.713424 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-envoy" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:42.713540 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.713520 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-token" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:42.713742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.713719 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-podinfo" pod "e01a4b7c-5d26-4dd8-847d-17b838f3c07d" (UID: "e01a4b7c-5d26-4dd8-847d-17b838f3c07d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:42.714036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.714017 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-kube-api-access-25qcs" pod "03dd63fa-31cb-488e-8232-b07a170bfc44" (UID: "03dd63fa-31cb-488e-8232-b07a170bfc44"). InnerVolumeSpecName "kube-api-access-25qcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:42.715405 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.715386 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-podinfo" pod "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" (UID: "b19c179c-d9bb-4a9b-bd05-7494fc8f01c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:42.807217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807180 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgnd\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-kube-api-access-8cgnd\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807220 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03dd63fa-31cb-488e-8232-b07a170bfc44-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807237 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807251 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807265 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807279 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807295 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807304 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807312 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807324 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807336 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807351 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807366 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807379 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807393 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807406 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807424 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97bws\" (UniqueName: \"kubernetes.io/projected/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-kube-api-access-97bws\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807438 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.807456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807452 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25qcs\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-kube-api-access-25qcs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807469 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807483 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807497 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807511 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807526 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807537 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807545 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a4b7c-5d26-4dd8-847d-17b838f3c07d-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.808275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.807552 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03dd63fa-31cb-488e-8232-b07a170bfc44-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:42.837389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.837348 5272 generic.go:356] "Generic (PLEG): container finished" podID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerID="c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1" exitCode=0 Nov 03 22:50:42.837580 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.837422 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" event={"ID":"ade70f21-d0ee-4c05-aec9-602d5f1e22eb","Type":"ContainerDied","Data":"c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1"} Nov 03 22:50:42.837990 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.837971 5272 scope.go:122] "RemoveContainer" containerID="c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1" Nov 03 22:50:42.839309 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.839289 5272 generic.go:356] "Generic (PLEG): container finished" podID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerID="3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8" exitCode=0 Nov 03 22:50:42.839420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.839357 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" Nov 03 22:50:42.839420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.839383 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" event={"ID":"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3","Type":"ContainerDied","Data":"3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8"} Nov 03 22:50:42.839529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.839440 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck" event={"ID":"b19c179c-d9bb-4a9b-bd05-7494fc8f01c3","Type":"ContainerDied","Data":"f602e4616a9d0cb5025a970d7375bdcf7d5c6ba62c1c7f3b6af020a19e8acd10"} Nov 03 22:50:42.839529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.839481 5272 scope.go:122] "RemoveContainer" containerID="3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8" Nov 03 22:50:42.840147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.840108 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:42.840234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.840207 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:42.840291 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.840256 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:50:42.847110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.846998 5272 generic.go:356] "Generic (PLEG): container finished" podID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerID="28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f" exitCode=0 Nov 03 22:50:42.847110 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.847086 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" Nov 03 22:50:42.847409 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.847381 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" event={"ID":"e01a4b7c-5d26-4dd8-847d-17b838f3c07d","Type":"ContainerDied","Data":"28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f"} Nov 03 22:50:42.847520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.847441 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9" event={"ID":"e01a4b7c-5d26-4dd8-847d-17b838f3c07d","Type":"ContainerDied","Data":"6f0307aa727c3446e412897fe64dc3b81a4ead2bd2514983b91ee57241516479"} Nov 03 22:50:42.854424 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.854388 5272 generic.go:356] "Generic (PLEG): container finished" podID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerID="eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b" exitCode=0 Nov 03 22:50:42.854484 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.854443 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" Nov 03 22:50:42.854522 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.854511 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" event={"ID":"03dd63fa-31cb-488e-8232-b07a170bfc44","Type":"ContainerDied","Data":"eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b"} Nov 03 22:50:42.854580 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.854557 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr" event={"ID":"03dd63fa-31cb-488e-8232-b07a170bfc44","Type":"ContainerDied","Data":"6d74d321c927566572a19d963d3a173b5b6712cee28d7e5f747ecd491c89fdea"} Nov 03 22:50:42.854987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.854967 5272 scope.go:122] "RemoveContainer" containerID="3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8" Nov 03 22:50:42.855249 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:50:42.855222 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8\": container with ID starting with 3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8 not found: ID does not exist" containerID="3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8" Nov 03 22:50:42.855325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.855257 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8"} err="failed to get container status \"3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8\": rpc error: code = NotFound desc = could not find container \"3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8\": container with ID starting with 3c28ca6dbc5ced920d571ce9feb0bca8a8bdeaf59458d2d50ec0233562b55ea8 not found: ID does not exist" Nov 03 22:50:42.855325 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.855280 5272 scope.go:122] "RemoveContainer" containerID="28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f" Nov 03 22:50:42.867481 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.867457 5272 scope.go:122] "RemoveContainer" containerID="28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f" Nov 03 22:50:42.867963 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:50:42.867778 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f\": container with ID starting with 28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f not found: ID does not exist" containerID="28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f" Nov 03 22:50:42.867963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.867853 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f"} err="failed to get container status \"28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f\": rpc error: code = NotFound desc = could not find container \"28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f\": container with ID starting with 28f99dcdb0d384733e9389a9e355b1e7550dcce89922192821ce7c923555244f not found: ID does not exist" Nov 03 22:50:42.867963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.867881 5272 scope.go:122] "RemoveContainer" containerID="eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b" Nov 03 22:50:42.878476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.878460 5272 scope.go:122] "RemoveContainer" containerID="eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b" Nov 03 22:50:42.878739 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:50:42.878722 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b\": container with ID starting with eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b not found: ID does not exist" containerID="eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b" Nov 03 22:50:42.878790 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:42.878753 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b"} err="failed to get container status \"eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b\": rpc error: code = NotFound desc = could not find container \"eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b\": container with ID starting with eaa634b7e92b9b96dfbecb4ff97a6df5e5c272733a3a900c6d01c0ad250a9d4b not found: ID does not exist" Nov 03 22:50:43.305090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.302550 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9"] Nov 03 22:50:43.314132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.309338 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-2qsk9"] Nov 03 22:50:43.328687 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.326270 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck"] Nov 03 22:50:43.347068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.347036 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-pfkck"] Nov 03 22:50:43.409366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.409315 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr"] Nov 03 22:50:43.415756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.415723 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-sxzfr"] Nov 03 22:50:43.495692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.495617 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" path="/var/lib/kubelet/pods/03dd63fa-31cb-488e-8232-b07a170bfc44/volumes" Nov 03 22:50:43.496124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.496108 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" path="/var/lib/kubelet/pods/b19c179c-d9bb-4a9b-bd05-7494fc8f01c3/volumes" Nov 03 22:50:43.496577 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.496563 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" path="/var/lib/kubelet/pods/e01a4b7c-5d26-4dd8-847d-17b838f3c07d/volumes" Nov 03 22:50:43.881092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:43.881061 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" event={"ID":"ade70f21-d0ee-4c05-aec9-602d5f1e22eb","Type":"ContainerStarted","Data":"96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187"} Nov 03 22:50:44.300825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:44.297776 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:50:44.300825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:44.298041 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:50:44.300825 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:44.298096 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:50:45.287384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:45.287347 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:50:45.287849 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:45.287423 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:50:46.288128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:46.288086 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:50:46.288541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:46.288155 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:50:47.287908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:47.287873 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:50:47.288096 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:47.287935 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:50:48.287930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:48.287885 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:50:48.288334 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:48.287956 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:50:49.294898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:49.289256 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 22:50:49.294898 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:49.289332 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 22:50:50.292303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:50.292214 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:50:50.292715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:50.292702 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:50:50.293432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:50.293403 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:50:52.686292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.686227 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t"] Nov 03 22:50:52.688217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.688180 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" Nov 03 22:50:52.688374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.688360 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" Nov 03 22:50:52.688491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.688479 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2a3d556b-898c-4567-96c0-a4278f64f24a" containerName="payload-pre-processing" Nov 03 22:50:52.688662 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.688646 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3d556b-898c-4567-96c0-a4278f64f24a" containerName="payload-pre-processing" Nov 03 22:50:52.688861 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.688845 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" Nov 03 22:50:52.688975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.688962 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" Nov 03 22:50:52.689107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689094 5272 cpu_manager.go:414] "containerMap: removing container" podUID="453aaffc-2a66-4e0a-8371-64755cd3c0d5" containerName="payload-processing" Nov 03 22:50:52.689214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689196 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453aaffc-2a66-4e0a-8371-64755cd3c0d5" containerName="payload-processing" Nov 03 22:50:52.689305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689297 5272 cpu_manager.go:414] "containerMap: removing container" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" Nov 03 22:50:52.689372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689362 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" Nov 03 22:50:52.689466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689456 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fe71c086-eea6-4678-a26f-95426eead8cc" containerName="revoke-keys" Nov 03 22:50:52.689541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689532 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe71c086-eea6-4678-a26f-95426eead8cc" containerName="revoke-keys" Nov 03 22:50:52.689832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689815 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3d556b-898c-4567-96c0-a4278f64f24a" containerName="payload-pre-processing" Nov 03 22:50:52.689946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.689936 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01a4b7c-5d26-4dd8-847d-17b838f3c07d" containerName="istio-proxy" Nov 03 22:50:52.690046 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.690036 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="453aaffc-2a66-4e0a-8371-64755cd3c0d5" containerName="payload-processing" Nov 03 22:50:52.690124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.690114 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="03dd63fa-31cb-488e-8232-b07a170bfc44" containerName="istio-proxy" Nov 03 22:50:52.693058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.693040 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe71c086-eea6-4678-a26f-95426eead8cc" containerName="revoke-keys" Nov 03 22:50:52.693229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.693216 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19c179c-d9bb-4a9b-bd05-7494fc8f01c3" containerName="istio-proxy" Nov 03 22:50:52.704483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.704459 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:52.763512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.762876 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t"] Nov 03 22:50:52.803827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.793157 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2627c5\"" Nov 03 22:50:52.803827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:52.793362 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2627c5-dockercfg-4btql\"" Nov 03 22:50:53.013416 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.008864 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f"] Nov 03 22:50:53.017399 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.014892 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcfs\" (UniqueName: \"kubernetes.io/projected/d0152dff-65b4-433c-a22a-9e049aa15d41-kube-api-access-jwcfs\") pod \"payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:53.017399 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.014981 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0152dff-65b4-433c-a22a-9e049aa15d41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:53.091920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.091672 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f"] Nov 03 22:50:53.091920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.091830 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.134911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.131990 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmz4\" (UniqueName: \"kubernetes.io/projected/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-kube-api-access-nwmz4\") pod \"payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.134911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.132058 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcfs\" (UniqueName: \"kubernetes.io/projected/d0152dff-65b4-433c-a22a-9e049aa15d41-kube-api-access-jwcfs\") pod \"payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:53.134911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.132094 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0152dff-65b4-433c-a22a-9e049aa15d41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:53.134911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.132151 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.134911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.133484 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0152dff-65b4-433c-a22a-9e049aa15d41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:53.190011 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.189982 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcfs\" (UniqueName: \"kubernetes.io/projected/d0152dff-65b4-433c-a22a-9e049aa15d41-kube-api-access-jwcfs\") pod \"payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:53.234065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.234030 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmz4\" (UniqueName: \"kubernetes.io/projected/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-kube-api-access-nwmz4\") pod \"payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.234254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.234155 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.234906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.234883 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.291487 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.291455 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmz4\" (UniqueName: \"kubernetes.io/projected/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-kube-api-access-nwmz4\") pod \"payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.364543 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.364507 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:53.443982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.443937 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:53.746726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.746635 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t"] Nov 03 22:50:53.752037 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:50:53.752003 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0152dff_65b4_433c_a22a_9e049aa15d41.slice/crio-a8204937577d838c03fde10aebd72592ab5fd804762483a55a206eb6f53568ae WatchSource:0}: Error finding container a8204937577d838c03fde10aebd72592ab5fd804762483a55a206eb6f53568ae: Status 404 returned error can't find the container with id a8204937577d838c03fde10aebd72592ab5fd804762483a55a206eb6f53568ae Nov 03 22:50:53.851545 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:53.851500 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f"] Nov 03 22:50:53.858247 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:50:53.858218 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176f3cd2_99cf_4a3a_a2f7_db6c0da7df63.slice/crio-26272c01d4f1676a378e34783b722311720ecc417cc3235679cdef1ac9284b9a WatchSource:0}: Error finding container 26272c01d4f1676a378e34783b722311720ecc417cc3235679cdef1ac9284b9a: Status 404 returned error can't find the container with id 26272c01d4f1676a378e34783b722311720ecc417cc3235679cdef1ac9284b9a Nov 03 22:50:54.107256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.107215 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" event={"ID":"d0152dff-65b4-433c-a22a-9e049aa15d41","Type":"ContainerStarted","Data":"6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196"} Nov 03 22:50:54.107467 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.107283 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" event={"ID":"d0152dff-65b4-433c-a22a-9e049aa15d41","Type":"ContainerStarted","Data":"a8204937577d838c03fde10aebd72592ab5fd804762483a55a206eb6f53568ae"} Nov 03 22:50:54.107467 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.107310 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:50:54.109069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.109043 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" event={"ID":"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63","Type":"ContainerStarted","Data":"884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686"} Nov 03 22:50:54.109188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.109083 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" event={"ID":"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63","Type":"ContainerStarted","Data":"26272c01d4f1676a378e34783b722311720ecc417cc3235679cdef1ac9284b9a"} Nov 03 22:50:54.109188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.109117 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:50:54.223448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.223363 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" podStartSLOduration=2.2233336870000002 podStartE2EDuration="2.223333687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:54.22207291 +0000 UTC m=+1634.844381221" watchObservedRunningTime="2026-11-03 22:50:54.223333687 +0000 UTC m=+1634.845641998" Nov 03 22:50:54.338177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:50:54.338113 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" podStartSLOduration=2.3380916689999998 podStartE2EDuration="2.338091669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:54.330998992 +0000 UTC m=+1634.953307306" watchObservedRunningTime="2026-11-03 22:50:54.338091669 +0000 UTC m=+1634.960399978" Nov 03 22:50:54.541350 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:50:54.541291 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:51:02.579827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.572286 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b"] Nov 03 22:51:02.807302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.807260 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:02.811676 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.811639 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b"] Nov 03 22:51:02.816562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.816528 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b"] Nov 03 22:51:02.823423 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.823398 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:02.823886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.823862 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b"] Nov 03 22:51:02.856741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.856705 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2420f6-dockercfg-p2rxb\"" Nov 03 22:51:02.856923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.856760 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2420f6\"" Nov 03 22:51:02.892909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.892876 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2lm7\" (UniqueName: \"kubernetes.io/projected/50c2cc70-8d48-420b-8265-23ce3c630947-kube-api-access-c2lm7\") pod \"payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:02.893082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.893061 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c2cc70-8d48-420b-8265-23ce3c630947-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:02.893148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.893094 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgq6\" (UniqueName: \"kubernetes.io/projected/9719dfa9-5c72-471f-8ded-b0f8b9031349-kube-api-access-5xgq6\") pod \"payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:02.893348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.893325 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9719dfa9-5c72-471f-8ded-b0f8b9031349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:02.995003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.994968 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2lm7\" (UniqueName: \"kubernetes.io/projected/50c2cc70-8d48-420b-8265-23ce3c630947-kube-api-access-c2lm7\") pod \"payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:02.995179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.995033 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c2cc70-8d48-420b-8265-23ce3c630947-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:02.995179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.995062 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgq6\" (UniqueName: \"kubernetes.io/projected/9719dfa9-5c72-471f-8ded-b0f8b9031349-kube-api-access-5xgq6\") pod \"payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:02.995322 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.995180 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9719dfa9-5c72-471f-8ded-b0f8b9031349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:02.995758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.995736 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c2cc70-8d48-420b-8265-23ce3c630947-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:02.995830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:02.995797 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9719dfa9-5c72-471f-8ded-b0f8b9031349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:03.052772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:03.052739 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgq6\" (UniqueName: \"kubernetes.io/projected/9719dfa9-5c72-471f-8ded-b0f8b9031349-kube-api-access-5xgq6\") pod \"payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:03.052934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:03.052861 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2lm7\" (UniqueName: \"kubernetes.io/projected/50c2cc70-8d48-420b-8265-23ce3c630947-kube-api-access-c2lm7\") pod \"payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:03.127828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:03.127742 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:03.139441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:03.139416 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:03.531408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:03.531331 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b"] Nov 03 22:51:03.577149 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:51:03.577122 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c2cc70_8d48_420b_8265_23ce3c630947.slice/crio-e12c2b8a9f45d309032cd87d6c839ff687ebc2c91310a383a717a5dd4215178a WatchSource:0}: Error finding container e12c2b8a9f45d309032cd87d6c839ff687ebc2c91310a383a717a5dd4215178a: Status 404 returned error can't find the container with id e12c2b8a9f45d309032cd87d6c839ff687ebc2c91310a383a717a5dd4215178a Nov 03 22:51:03.583468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:03.583439 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b"] Nov 03 22:51:03.590739 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:51:03.590710 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9719dfa9_5c72_471f_8ded_b0f8b9031349.slice/crio-b74dc6edff4f6b7b24b7875c3d901d6c6b0e923a9ccfaddedcb3673a7fb1a9e9 WatchSource:0}: Error finding container b74dc6edff4f6b7b24b7875c3d901d6c6b0e923a9ccfaddedcb3673a7fb1a9e9: Status 404 returned error can't find the container with id b74dc6edff4f6b7b24b7875c3d901d6c6b0e923a9ccfaddedcb3673a7fb1a9e9 Nov 03 22:51:04.272943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.272905 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" event={"ID":"9719dfa9-5c72-471f-8ded-b0f8b9031349","Type":"ContainerStarted","Data":"aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c"} Nov 03 22:51:04.273143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.272968 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" event={"ID":"9719dfa9-5c72-471f-8ded-b0f8b9031349","Type":"ContainerStarted","Data":"b74dc6edff4f6b7b24b7875c3d901d6c6b0e923a9ccfaddedcb3673a7fb1a9e9"} Nov 03 22:51:04.273143 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.272999 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:04.274695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.274669 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" event={"ID":"50c2cc70-8d48-420b-8265-23ce3c630947","Type":"ContainerStarted","Data":"ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8"} Nov 03 22:51:04.274842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.274717 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" event={"ID":"50c2cc70-8d48-420b-8265-23ce3c630947","Type":"ContainerStarted","Data":"e12c2b8a9f45d309032cd87d6c839ff687ebc2c91310a383a717a5dd4215178a"} Nov 03 22:51:04.274842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.274744 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:04.388564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.388512 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" podStartSLOduration=2.388498173 podStartE2EDuration="2.388498173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:04.385106803 +0000 UTC m=+1645.007415115" watchObservedRunningTime="2026-11-03 22:51:04.388498173 +0000 UTC m=+1645.010806506" Nov 03 22:51:04.494265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:04.494216 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" podStartSLOduration=2.494203596 podStartE2EDuration="2.494203596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:04.493397218 +0000 UTC m=+1645.115705525" watchObservedRunningTime="2026-11-03 22:51:04.494203596 +0000 UTC m=+1645.116511903" Nov 03 22:51:05.125888 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:05.124677 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:51:05.154839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:05.154022 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:51:14.117864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:14.117797 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:51:15.291478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:15.291449 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:51:15.292534 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:15.292512 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:51:24.507282 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:51:24.507233 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:51:26.698374 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.698341 5272 generic.go:356] "Generic (PLEG): container finished" podID="45585d1a-823c-4006-bb93-f257c3dcb895" containerID="9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf" exitCode=0 Nov 03 22:51:26.698843 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.698412 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" event={"ID":"45585d1a-823c-4006-bb93-f257c3dcb895","Type":"ContainerDied","Data":"9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf"} Nov 03 22:51:26.698969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.698949 5272 scope.go:122] "RemoveContainer" containerID="9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf" Nov 03 22:51:26.700398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.700374 5272 generic.go:356] "Generic (PLEG): container finished" podID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerID="f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb" exitCode=0 Nov 03 22:51:26.700478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.700420 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" event={"ID":"d448175a-3e0e-4b5d-b786-010d6a58a1af","Type":"ContainerDied","Data":"f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb"} Nov 03 22:51:26.700759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.700735 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:26.700879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.700790 5272 scope.go:122] "RemoveContainer" containerID="f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb" Nov 03 22:51:26.700879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.700833 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:26.700879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.700875 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:26.702346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.702318 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:26.702416 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.702382 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:26.702455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:26.702417 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:51:27.712889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:27.712853 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" event={"ID":"d448175a-3e0e-4b5d-b786-010d6a58a1af","Type":"ContainerStarted","Data":"ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2"} Nov 03 22:51:27.715216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:27.715189 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" event={"ID":"45585d1a-823c-4006-bb93-f257c3dcb895","Type":"ContainerStarted","Data":"e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb"} Nov 03 22:51:27.732652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:27.732621 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:51:27.732966 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:27.732942 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:51:27.733036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:27.733017 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:51:28.197104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:28.196877 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:51:28.198710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:28.198674 5272 patch_prober.go:29] interesting pod/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:51:28.198910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:28.198740 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:51:28.817826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:28.811026 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:51:28.817826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:28.811110 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:51:29.044889 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.044851 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42"] Nov 03 22:51:29.072890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.067839 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.118235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.118199 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-922795\"" Nov 03 22:51:29.124835 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.118734 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-922795-dockercfg-wrr87\"" Nov 03 22:51:29.199830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.178367 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42"] Nov 03 22:51:29.309947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.267583 5272 patch_prober.go:29] interesting pod/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:51:29.310229 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.310199 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:51:29.327986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.288751 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3927a87-6644-42c5-a642-e7ec788f7e43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.328328 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.328298 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmtw\" (UniqueName: \"kubernetes.io/projected/d3927a87-6644-42c5-a642-e7ec788f7e43-kube-api-access-6jmtw\") pod \"payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.441833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.432942 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3927a87-6644-42c5-a642-e7ec788f7e43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.441833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.433025 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmtw\" (UniqueName: \"kubernetes.io/projected/d3927a87-6644-42c5-a642-e7ec788f7e43-kube-api-access-6jmtw\") pod \"payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.441833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.434570 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3927a87-6644-42c5-a642-e7ec788f7e43-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.588022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.587911 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:51:29.641020 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.640972 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:51:29.641983 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.641948 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmtw\" (UniqueName: \"kubernetes.io/projected/d3927a87-6644-42c5-a642-e7ec788f7e43-kube-api-access-6jmtw\") pod \"payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.681125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.680971 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:51:29.709468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.709432 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq"] Nov 03 22:51:29.721357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.721332 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq"] Nov 03 22:51:29.721546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.721415 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:29.732591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.732559 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:51:29.732709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.732631 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:51:29.741490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.741468 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:29.811022 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.810981 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d3ddf0-3385-463d-9c06-1c79fd8faa57-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:29.811256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.811108 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26kw\" (UniqueName: \"kubernetes.io/projected/35d3ddf0-3385-463d-9c06-1c79fd8faa57-kube-api-access-g26kw\") pod \"payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:29.891145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.891035 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:51:29.925362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.913260 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d3ddf0-3385-463d-9c06-1c79fd8faa57-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:29.925362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.913343 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26kw\" (UniqueName: \"kubernetes.io/projected/35d3ddf0-3385-463d-9c06-1c79fd8faa57-kube-api-access-g26kw\") pod \"payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:29.925362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:29.914587 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d3ddf0-3385-463d-9c06-1c79fd8faa57-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:30.128664 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.128587 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26kw\" (UniqueName: \"kubernetes.io/projected/35d3ddf0-3385-463d-9c06-1c79fd8faa57-kube-api-access-g26kw\") pod \"payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:30.201149 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.201015 5272 patch_prober.go:29] interesting pod/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:51:30.201547 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.201438 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:51:30.337517 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.337472 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:30.501930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.501492 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42"] Nov 03 22:51:30.506344 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:51:30.506301 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3927a87_6644_42c5_a642_e7ec788f7e43.slice/crio-7a848928b2bbf3f84c03820c35c943407586a02a0500afd06e0c9d02bc77ba09 WatchSource:0}: Error finding container 7a848928b2bbf3f84c03820c35c943407586a02a0500afd06e0c9d02bc77ba09: Status 404 returned error can't find the container with id 7a848928b2bbf3f84c03820c35c943407586a02a0500afd06e0c9d02bc77ba09 Nov 03 22:51:30.733024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.732988 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:51:30.733198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.733040 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:51:30.826922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.826872 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq"] Nov 03 22:51:30.832063 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:51:30.832031 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d3ddf0_3385_463d_9c06_1c79fd8faa57.slice/crio-febe8efee8f7ec74e90c9cf76feeac39a845099ad18e79c10bca6d6831149dc4 WatchSource:0}: Error finding container febe8efee8f7ec74e90c9cf76feeac39a845099ad18e79c10bca6d6831149dc4: Status 404 returned error can't find the container with id febe8efee8f7ec74e90c9cf76feeac39a845099ad18e79c10bca6d6831149dc4 Nov 03 22:51:30.921519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.921474 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" event={"ID":"d3927a87-6644-42c5-a642-e7ec788f7e43","Type":"ContainerStarted","Data":"9e051dee4ff804fc5e2237d88f466b7ad6a7abe5a3a7ba2bfb47c83b5b412183"} Nov 03 22:51:30.922006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.921539 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" event={"ID":"d3927a87-6644-42c5-a642-e7ec788f7e43","Type":"ContainerStarted","Data":"7a848928b2bbf3f84c03820c35c943407586a02a0500afd06e0c9d02bc77ba09"} Nov 03 22:51:30.922006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.921766 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:30.922652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:30.922629 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" event={"ID":"35d3ddf0-3385-463d-9c06-1c79fd8faa57","Type":"ContainerStarted","Data":"febe8efee8f7ec74e90c9cf76feeac39a845099ad18e79c10bca6d6831149dc4"} Nov 03 22:51:31.036060 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:31.036004 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" podStartSLOduration=3.035988164 podStartE2EDuration="3.035988164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:31.033038289 +0000 UTC m=+1671.655346590" watchObservedRunningTime="2026-11-03 22:51:31.035988164 +0000 UTC m=+1671.658296501" Nov 03 22:51:31.200873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:31.200281 5272 patch_prober.go:29] interesting pod/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:51:31.200873 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:31.200350 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:51:31.740029 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:31.739988 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:51:31.740342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:31.740303 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:51:31.932214 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:31.932182 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" event={"ID":"35d3ddf0-3385-463d-9c06-1c79fd8faa57","Type":"ContainerStarted","Data":"1384a42c43fb1fa3b90b701fd6f9a824e3c89277c4adfa941c7151ebb9d12b63"} Nov 03 22:51:31.932710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:31.932694 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:32.095549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:32.083121 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" podStartSLOduration=3.083098716 podStartE2EDuration="3.083098716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:32.047165698 +0000 UTC m=+1672.669474007" watchObservedRunningTime="2026-11-03 22:51:32.083098716 +0000 UTC m=+1672.705407025" Nov 03 22:51:32.204827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:32.204456 5272 patch_prober.go:29] interesting pod/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:51:32.204827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:32.204514 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:51:32.772886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:32.757059 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:51:32.772886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:32.761950 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:51:33.293828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:33.281452 5272 patch_prober.go:29] interesting pod/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:51:33.293828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:33.281593 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 03 22:51:33.732946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:33.732856 5272 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 22:51:33.732946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:33.732910 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 22:51:34.201117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:34.201090 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:51:34.202456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:34.202425 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:51:34.737210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:34.737180 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:51:34.737694 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:34.737680 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:51:34.738454 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:34.738430 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:51:41.936327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:41.936295 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:51:42.523739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:42.523701 5272 scope.go:122] "RemoveContainer" containerID="265391b72ef7afe97b4ee398cda8b61a47326bc82ea482bb89950f8b680c236c" Nov 03 22:51:43.335136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:43.335107 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:51:43.912206 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:43.912149 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs"] Nov 03 22:51:43.949097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:43.949063 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:43.965742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:43.965697 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs"] Nov 03 22:51:44.010057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.010022 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-65913e-dockercfg-52ghd\"" Nov 03 22:51:44.030828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.011122 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-65913e\"" Nov 03 22:51:44.277383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.277294 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e894261-faf0-4e8a-8d88-ffb419dae171-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:44.277775 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.277754 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4bf\" (UniqueName: \"kubernetes.io/projected/8e894261-faf0-4e8a-8d88-ffb419dae171-kube-api-access-7v4bf\") pod \"payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:44.298600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.298548 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45"] Nov 03 22:51:44.313072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.313033 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:44.319642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.319598 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45"] Nov 03 22:51:44.379504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.379468 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e894261-faf0-4e8a-8d88-ffb419dae171-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:44.379943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.379619 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97d1225-8a29-40fb-8c43-3f810ad551c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:44.379943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.379659 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4bf\" (UniqueName: \"kubernetes.io/projected/8e894261-faf0-4e8a-8d88-ffb419dae171-kube-api-access-7v4bf\") pod \"payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:44.379943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.379879 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllx4\" (UniqueName: \"kubernetes.io/projected/c97d1225-8a29-40fb-8c43-3f810ad551c2-kube-api-access-vllx4\") pod \"payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:44.380331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.380304 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e894261-faf0-4e8a-8d88-ffb419dae171-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:44.437394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.437357 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4bf\" (UniqueName: \"kubernetes.io/projected/8e894261-faf0-4e8a-8d88-ffb419dae171-kube-api-access-7v4bf\") pod \"payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:44.481926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.481897 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97d1225-8a29-40fb-8c43-3f810ad551c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:44.482124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.482009 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllx4\" (UniqueName: \"kubernetes.io/projected/c97d1225-8a29-40fb-8c43-3f810ad551c2-kube-api-access-vllx4\") pod \"payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:44.482620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.482596 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97d1225-8a29-40fb-8c43-3f810ad551c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:44.539627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.539600 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllx4\" (UniqueName: \"kubernetes.io/projected/c97d1225-8a29-40fb-8c43-3f810ad551c2-kube-api-access-vllx4\") pod \"payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:44.586839 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.586786 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:44.634308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:44.634274 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:45.061826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.059688 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs"] Nov 03 22:51:45.191593 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.191549 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45"] Nov 03 22:51:45.196973 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:51:45.196930 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97d1225_8a29_40fb_8c43_3f810ad551c2.slice/crio-f8ad8e4aa9c2d903098368a5ee0fa2a6fc27972e4b16d23575a48430479f41f6 WatchSource:0}: Error finding container f8ad8e4aa9c2d903098368a5ee0fa2a6fc27972e4b16d23575a48430479f41f6: Status 404 returned error can't find the container with id f8ad8e4aa9c2d903098368a5ee0fa2a6fc27972e4b16d23575a48430479f41f6 Nov 03 22:51:45.496202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.496125 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:45.496202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.496174 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:45.496202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.496193 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" event={"ID":"c97d1225-8a29-40fb-8c43-3f810ad551c2","Type":"ContainerStarted","Data":"ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e"} Nov 03 22:51:45.496688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.496211 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" event={"ID":"c97d1225-8a29-40fb-8c43-3f810ad551c2","Type":"ContainerStarted","Data":"f8ad8e4aa9c2d903098368a5ee0fa2a6fc27972e4b16d23575a48430479f41f6"} Nov 03 22:51:45.496688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.496227 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" event={"ID":"8e894261-faf0-4e8a-8d88-ffb419dae171","Type":"ContainerStarted","Data":"a240c58cf3d2a5e5474d290b8ee6feafa2c69cfe57f4a6d37aaf715867686460"} Nov 03 22:51:45.496688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.496242 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" event={"ID":"8e894261-faf0-4e8a-8d88-ffb419dae171","Type":"ContainerStarted","Data":"e9c77f66fbe48ace863a963ab3ede91c4bf08fa803b4767700c3485c3c516e59"} Nov 03 22:51:45.605510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.605453 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" podStartSLOduration=1.6054385880000002 podStartE2EDuration="1.605438588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:45.602153266 +0000 UTC m=+1686.224461574" watchObservedRunningTime="2026-11-03 22:51:45.605438588 +0000 UTC m=+1686.227746895" Nov 03 22:51:45.715461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:45.715399 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" podStartSLOduration=2.715380097 podStartE2EDuration="2.715380097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:45.712087441 +0000 UTC m=+1686.334395749" watchObservedRunningTime="2026-11-03 22:51:45.715380097 +0000 UTC m=+1686.337688405" Nov 03 22:51:46.034485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:46.034448 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d"] Nov 03 22:51:46.034782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:46.034663 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" containerID="cri-o://81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c" gracePeriod=30 Nov 03 22:51:51.322661 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.322631 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:51:51.443532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443490 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-podinfo\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.443691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443588 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-data\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.443740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443697 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-certs\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.443740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443726 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-socket\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.443887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443857 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-envoy\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.443963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443948 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5b706b-d8dc-4e29-a625-65739631a25c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5b706b-d8dc-4e29-a625-65739631a25c-istiod-ca-cert\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.444006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443988 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-token\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.444006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.443986 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-data" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:51.444113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444003 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-certs" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:51.444113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444025 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-kube-api-access-fr6vm\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-kube-api-access-fr6vm\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.444113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444064 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-credential-socket\") pod \"2b5b706b-d8dc-4e29-a625-65739631a25c\" (UID: \"2b5b706b-d8dc-4e29-a625-65739631a25c\") " Nov 03 22:51:51.444113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444076 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-socket" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:51.444388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444360 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-credential-socket" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:51.444509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444412 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5b706b-d8dc-4e29-a625-65739631a25c-istiod-ca-cert" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:51.444972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444950 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.445044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444981 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.445044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.444997 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5b706b-d8dc-4e29-a625-65739631a25c-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.445044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.445011 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.445044 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.445026 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.446314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.446288 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-kube-api-access-fr6vm" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "kube-api-access-fr6vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:51.446409 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.446319 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-podinfo" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:51.446479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.446464 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-token" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:51.446479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.446468 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-envoy" pod "2b5b706b-d8dc-4e29-a625-65739631a25c" (UID: "2b5b706b-d8dc-4e29-a625-65739631a25c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:51.546362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.546317 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.546362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.546347 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.546362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.546357 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6vm\" (UniqueName: \"kubernetes.io/projected/2b5b706b-d8dc-4e29-a625-65739631a25c-kube-api-access-fr6vm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.546362 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.546367 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5b706b-d8dc-4e29-a625-65739631a25c-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.557075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.556997 5272 generic.go:356] "Generic (PLEG): container finished" podID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerID="81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c" exitCode=0 Nov 03 22:51:51.557075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.557043 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" Nov 03 22:51:51.557075 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.557054 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" event={"ID":"2b5b706b-d8dc-4e29-a625-65739631a25c","Type":"ContainerDied","Data":"81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c"} Nov 03 22:51:51.557303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.557097 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d" event={"ID":"2b5b706b-d8dc-4e29-a625-65739631a25c","Type":"ContainerDied","Data":"5d77971fdb8cff16044587ca73fd18edff2d21faea22bcfe404848425e2b08ab"} Nov 03 22:51:51.557303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.557118 5272 scope.go:122] "RemoveContainer" containerID="81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c" Nov 03 22:51:51.568235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.568218 5272 scope.go:122] "RemoveContainer" containerID="81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c" Nov 03 22:51:51.568528 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:51:51.568508 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c\": container with ID starting with 81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c not found: ID does not exist" containerID="81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c" Nov 03 22:51:51.568586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.568537 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c"} err="failed to get container status \"81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c\": rpc error: code = NotFound desc = could not find container \"81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c\": container with ID starting with 81b7dab434272e41d49e14f0cbb02c46408fccc81d5716801772fc00421b062c not found: ID does not exist" Nov 03 22:51:51.714006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.713965 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d"] Nov 03 22:51:51.717704 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:51.717672 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-k5m5d"] Nov 03 22:51:53.495536 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:53.495491 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" path="/var/lib/kubelet/pods/2b5b706b-d8dc-4e29-a625-65739631a25c/volumes" Nov 03 22:51:53.579734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:53.579696 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh"] Nov 03 22:51:53.580052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:53.580018 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" containerID="cri-o://f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24" gracePeriod=30 Nov 03 22:51:54.503330 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:51:54.503290 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:51:56.506617 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:56.506580 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:51:57.523868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:57.523833 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:51:58.293666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.293623 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb"] Nov 03 22:51:58.295911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.295869 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" Nov 03 22:51:58.296106 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.296092 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" Nov 03 22:51:58.296589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.296558 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5b706b-d8dc-4e29-a625-65739631a25c" containerName="istio-proxy" Nov 03 22:51:58.311666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.311635 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.379604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.379558 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb"] Nov 03 22:51:58.410855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.392968 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-0189ef\"" Nov 03 22:51:58.426420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.426384 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpw4\" (UniqueName: \"kubernetes.io/projected/2237d33f-a103-4ad7-908a-abaa7731658c-kube-api-access-5bpw4\") pod \"payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.433826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.426902 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2237d33f-a103-4ad7-908a-abaa7731658c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.577187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.566079 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2237d33f-a103-4ad7-908a-abaa7731658c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.577187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.566371 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpw4\" (UniqueName: \"kubernetes.io/projected/2237d33f-a103-4ad7-908a-abaa7731658c-kube-api-access-5bpw4\") pod \"payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.577187 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.567974 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2237d33f-a103-4ad7-908a-abaa7731658c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.695987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.679175 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpw4\" (UniqueName: \"kubernetes.io/projected/2237d33f-a103-4ad7-908a-abaa7731658c-kube-api-access-5bpw4\") pod \"payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.695987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.689333 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j"] Nov 03 22:51:58.710787 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.710755 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:58.756166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.756004 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j"] Nov 03 22:51:58.756166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.756124 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:58.805311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.805274 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-0189ef-dockercfg-gk7r2\"" Nov 03 22:51:58.828997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.828928 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzlbm\" (UniqueName: \"kubernetes.io/projected/6bb5cb89-168d-4eac-863d-a769432088fe-kube-api-access-qzlbm\") pod \"payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:58.829202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.829180 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb5cb89-168d-4eac-863d-a769432088fe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:58.931260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.931218 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzlbm\" (UniqueName: \"kubernetes.io/projected/6bb5cb89-168d-4eac-863d-a769432088fe-kube-api-access-qzlbm\") pod \"payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:58.931460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.931440 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb5cb89-168d-4eac-863d-a769432088fe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:58.932582 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.932552 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb5cb89-168d-4eac-863d-a769432088fe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:58.957213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.957189 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:51:58.988858 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:58.988825 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzlbm\" (UniqueName: \"kubernetes.io/projected/6bb5cb89-168d-4eac-863d-a769432088fe-kube-api-access-qzlbm\") pod \"payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:59.032874 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.032834 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-envoy\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.033054 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.032964 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-credential-socket\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.033148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.033127 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-data\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.033504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.033404 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-credential-socket" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:59.033757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.033728 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-data" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:59.034672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.034649 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-socket\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.034770 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.034721 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c945d247-a300-4b2c-bf34-a46626323f6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c945d247-a300-4b2c-bf34-a46626323f6e-istiod-ca-cert\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.034862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.034829 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-istio-token\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-istio-token\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.035109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.035086 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-socket" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:59.035197 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.035092 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-certs\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.035197 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.035153 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c945d247-a300-4b2c-bf34-a46626323f6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c945d247-a300-4b2c-bf34-a46626323f6e-istio-podinfo\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.035318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.035197 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-kube-api-access-zr78t\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-kube-api-access-zr78t\") pod \"c945d247-a300-4b2c-bf34-a46626323f6e\" (UID: \"c945d247-a300-4b2c-bf34-a46626323f6e\") " Nov 03 22:51:59.035318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.035266 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-certs" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:59.035462 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.035439 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c945d247-a300-4b2c-bf34-a46626323f6e-istiod-ca-cert" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:59.035546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.035465 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-envoy" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:59.036039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.036014 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.036135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.036043 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.036135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.036058 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.036135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.036072 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.036135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.036085 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c945d247-a300-4b2c-bf34-a46626323f6e-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.036135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.036099 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c945d247-a300-4b2c-bf34-a46626323f6e-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.037405 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.037378 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-istio-token" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:59.037508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.037461 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c945d247-a300-4b2c-bf34-a46626323f6e-istio-podinfo" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:59.037591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.037572 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-kube-api-access-zr78t" pod "c945d247-a300-4b2c-bf34-a46626323f6e" (UID: "c945d247-a300-4b2c-bf34-a46626323f6e"). InnerVolumeSpecName "kube-api-access-zr78t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:59.106250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.106165 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:59.137225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.137198 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.137225 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.137226 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c945d247-a300-4b2c-bf34-a46626323f6e-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.137386 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.137237 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr78t\" (UniqueName: \"kubernetes.io/projected/c945d247-a300-4b2c-bf34-a46626323f6e-kube-api-access-zr78t\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.221082 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.221053 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb"] Nov 03 22:51:59.224283 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:51:59.224248 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2237d33f_a103_4ad7_908a_abaa7731658c.slice/crio-13b5958629595b9834658a936e4fedc8cafe3217bf3aff3321342d566d3637f8 WatchSource:0}: Error finding container 13b5958629595b9834658a936e4fedc8cafe3217bf3aff3321342d566d3637f8: Status 404 returned error can't find the container with id 13b5958629595b9834658a936e4fedc8cafe3217bf3aff3321342d566d3637f8 Nov 03 22:51:59.495134 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:51:59.495103 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb5cb89_168d_4eac_863d_a769432088fe.slice/crio-c6ef96cf618f4b98b66ded53fa76dd931a05aea05abde2be7018196dda9da3d8 WatchSource:0}: Error finding container c6ef96cf618f4b98b66ded53fa76dd931a05aea05abde2be7018196dda9da3d8: Status 404 returned error can't find the container with id c6ef96cf618f4b98b66ded53fa76dd931a05aea05abde2be7018196dda9da3d8 Nov 03 22:51:59.499260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.499146 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j"] Nov 03 22:51:59.734921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.734873 5272 generic.go:356] "Generic (PLEG): container finished" podID="c945d247-a300-4b2c-bf34-a46626323f6e" containerID="f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24" exitCode=0 Nov 03 22:51:59.735717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.734959 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" Nov 03 22:51:59.735717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.735003 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" event={"ID":"c945d247-a300-4b2c-bf34-a46626323f6e","Type":"ContainerDied","Data":"f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24"} Nov 03 22:51:59.735717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.735089 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh" event={"ID":"c945d247-a300-4b2c-bf34-a46626323f6e","Type":"ContainerDied","Data":"99ee8fcd3dc531f0e7a4f8d72d10769ffda20634b264be78fd477a162b63d1c9"} Nov 03 22:51:59.735717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.735121 5272 scope.go:122] "RemoveContainer" containerID="f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24" Nov 03 22:51:59.739077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.739039 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" event={"ID":"6bb5cb89-168d-4eac-863d-a769432088fe","Type":"ContainerStarted","Data":"f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94"} Nov 03 22:51:59.739190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.739097 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" event={"ID":"6bb5cb89-168d-4eac-863d-a769432088fe","Type":"ContainerStarted","Data":"c6ef96cf618f4b98b66ded53fa76dd931a05aea05abde2be7018196dda9da3d8"} Nov 03 22:51:59.739507 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.739486 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:51:59.741528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.741456 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" event={"ID":"2237d33f-a103-4ad7-908a-abaa7731658c","Type":"ContainerStarted","Data":"65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0"} Nov 03 22:51:59.741528 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.741510 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:51:59.741756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.741533 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" event={"ID":"2237d33f-a103-4ad7-908a-abaa7731658c","Type":"ContainerStarted","Data":"13b5958629595b9834658a936e4fedc8cafe3217bf3aff3321342d566d3637f8"} Nov 03 22:51:59.750221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.750201 5272 scope.go:122] "RemoveContainer" containerID="f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24" Nov 03 22:51:59.750551 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:51:59.750526 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24\": container with ID starting with f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24 not found: ID does not exist" containerID="f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24" Nov 03 22:51:59.750640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.750560 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24"} err="failed to get container status \"f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24\": rpc error: code = NotFound desc = could not find container \"f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24\": container with ID starting with f36b0c2f52b98a5fa8b8963da16561d58c9b7d1339d47a93001e5503f4eddf24 not found: ID does not exist" Nov 03 22:51:59.901913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:51:59.901862 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" podStartSLOduration=1.90185034 podStartE2EDuration="1.90185034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:59.899162229 +0000 UTC m=+1700.521470536" watchObservedRunningTime="2026-11-03 22:51:59.90185034 +0000 UTC m=+1700.524158644" Nov 03 22:52:00.007507 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:00.007399 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" podStartSLOduration=2.007384931 podStartE2EDuration="2.007384931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:00.006609254 +0000 UTC m=+1700.628917562" watchObservedRunningTime="2026-11-03 22:52:00.007384931 +0000 UTC m=+1700.629693245" Nov 03 22:52:00.114884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:00.114846 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh"] Nov 03 22:52:00.120071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:00.120036 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-p8vxh"] Nov 03 22:52:01.531541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:01.531497 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" path="/var/lib/kubelet/pods/c945d247-a300-4b2c-bf34-a46626323f6e/volumes" Nov 03 22:52:09.037181 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.037139 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2"] Nov 03 22:52:09.038169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.038147 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" Nov 03 22:52:09.038169 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.038170 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" Nov 03 22:52:09.038312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.038301 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c945d247-a300-4b2c-bf34-a46626323f6e" containerName="istio-proxy" Nov 03 22:52:09.043514 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.043496 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.095195 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.095163 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2"] Nov 03 22:52:09.106749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.106707 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzv4z\" (UniqueName: \"kubernetes.io/projected/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-kube-api-access-zzv4z\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.106958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.106784 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.107035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.106978 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.136519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.136485 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:52:09.136691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.136552 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:52:09.138632 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.138613 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:09.138738 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.138725 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:52:09.208741 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.208698 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzv4z\" (UniqueName: \"kubernetes.io/projected/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-kube-api-access-zzv4z\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.208937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.208775 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.208993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.208958 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.209485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.209464 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.211358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.211334 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.266381 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.266350 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzv4z\" (UniqueName: \"kubernetes.io/projected/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-kube-api-access-zzv4z\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-hkkr2\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.358282 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.358191 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:09.743117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.743040 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2"] Nov 03 22:52:09.748468 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:52:09.748434 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddedda6_fe9e_412b_b5a4_74a46141a6b8.slice/crio-b0406c120fd96294481750048d08614bb2177dec9b5f03aff7be95828f42bdf1 WatchSource:0}: Error finding container b0406c120fd96294481750048d08614bb2177dec9b5f03aff7be95828f42bdf1: Status 404 returned error can't find the container with id b0406c120fd96294481750048d08614bb2177dec9b5f03aff7be95828f42bdf1 Nov 03 22:52:09.853227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:09.853189 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" event={"ID":"6ddedda6-fe9e-412b-b5a4-74a46141a6b8","Type":"ContainerStarted","Data":"b0406c120fd96294481750048d08614bb2177dec9b5f03aff7be95828f42bdf1"} Nov 03 22:52:10.754388 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:10.754354 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:52:10.754865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:10.754425 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:52:10.863078 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:10.863044 5272 generic.go:356] "Generic (PLEG): container finished" podID="6ddedda6-fe9e-412b-b5a4-74a46141a6b8" containerID="ed7167abc2a5d51858d4a9df94b165739fd537e213b55cd02edd1ce5b7801015" exitCode=0 Nov 03 22:52:10.863270 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:10.863124 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" event={"ID":"6ddedda6-fe9e-412b-b5a4-74a46141a6b8","Type":"ContainerDied","Data":"ed7167abc2a5d51858d4a9df94b165739fd537e213b55cd02edd1ce5b7801015"} Nov 03 22:52:12.052534 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.052500 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:12.101393 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.101361 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-openshift-ca\") pod \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " Nov 03 22:52:12.101583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.101425 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-xks-ca\") pod \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " Nov 03 22:52:12.101583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.101479 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-kube-api-access-zzv4z\" (UniqueName: \"kubernetes.io/projected/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-kube-api-access-zzv4z\") pod \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\" (UID: \"6ddedda6-fe9e-412b-b5a4-74a46141a6b8\") " Nov 03 22:52:12.101860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.101831 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-openshift-ca" pod "6ddedda6-fe9e-412b-b5a4-74a46141a6b8" (UID: "6ddedda6-fe9e-412b-b5a4-74a46141a6b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:12.102030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.102010 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.103584 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.103553 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-xks-ca" pod "6ddedda6-fe9e-412b-b5a4-74a46141a6b8" (UID: "6ddedda6-fe9e-412b-b5a4-74a46141a6b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:12.103762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.103646 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-kube-api-access-zzv4z" pod "6ddedda6-fe9e-412b-b5a4-74a46141a6b8" (UID: "6ddedda6-fe9e-412b-b5a4-74a46141a6b8"). InnerVolumeSpecName "kube-api-access-zzv4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:12.204394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.204287 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.204394 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.204329 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzv4z\" (UniqueName: \"kubernetes.io/projected/6ddedda6-fe9e-412b-b5a4-74a46141a6b8-kube-api-access-zzv4z\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.880688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.880655 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" Nov 03 22:52:12.880904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.880689 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2" event={"ID":"6ddedda6-fe9e-412b-b5a4-74a46141a6b8","Type":"ContainerDied","Data":"b0406c120fd96294481750048d08614bb2177dec9b5f03aff7be95828f42bdf1"} Nov 03 22:52:12.880904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:12.880739 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0406c120fd96294481750048d08614bb2177dec9b5f03aff7be95828f42bdf1" Nov 03 22:52:21.761691 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:21.761602 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:52:22.917319 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:22.917279 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk"] Nov 03 22:52:23.028709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:23.028663 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k"] Nov 03 22:52:24.715901 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:52:24.706997 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:52:27.918640 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:27.918585 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" podUID="0ff84017-6a02-412b-aa89-8e1bc502fc84" containerName="payload-processing" containerID="cri-o://0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105" gracePeriod=25 Nov 03 22:52:28.030312 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.030257 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" podUID="641a4a4c-cce4-49e8-bedd-1977115c8325" containerName="payload-pre-processing" containerID="cri-o://2ebbf6c29beaa6275254068335e482a2cfeb0fcda857bfc130fa636943b50db3" gracePeriod=25 Nov 03 22:52:28.312826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.307561 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:52:28.342410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.342371 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" event={"ID":"0ff84017-6a02-412b-aa89-8e1bc502fc84","Type":"ContainerDied","Data":"0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105"} Nov 03 22:52:28.342579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.342457 5272 scope.go:122] "RemoveContainer" containerID="0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105" Nov 03 22:52:28.342765 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.342159 5272 generic.go:356] "Generic (PLEG): container finished" podID="0ff84017-6a02-412b-aa89-8e1bc502fc84" containerID="0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105" exitCode=0 Nov 03 22:52:28.343290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.343264 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" event={"ID":"0ff84017-6a02-412b-aa89-8e1bc502fc84","Type":"ContainerDied","Data":"c20efd31796270b9543ef2f8fcef6a8926d72e2b14d6af2e7745733f39d57166"} Nov 03 22:52:28.349692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.349649 5272 generic.go:356] "Generic (PLEG): container finished" podID="641a4a4c-cce4-49e8-bedd-1977115c8325" containerID="2ebbf6c29beaa6275254068335e482a2cfeb0fcda857bfc130fa636943b50db3" exitCode=0 Nov 03 22:52:28.350155 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.350133 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" event={"ID":"641a4a4c-cce4-49e8-bedd-1977115c8325","Type":"ContainerDied","Data":"2ebbf6c29beaa6275254068335e482a2cfeb0fcda857bfc130fa636943b50db3"} Nov 03 22:52:28.365283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.365015 5272 scope.go:122] "RemoveContainer" containerID="0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105" Nov 03 22:52:28.366276 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:52:28.366208 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105\": container with ID starting with 0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105 not found: ID does not exist" containerID="0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105" Nov 03 22:52:28.366276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.366248 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105"} err="failed to get container status \"0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105\": rpc error: code = NotFound desc = could not find container \"0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105\": container with ID starting with 0021e19a8c33bae084b9ddb577a4af66dd426dcda29fc42d4a587fe139ced105 not found: ID does not exist" Nov 03 22:52:28.366598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.366538 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff84017-6a02-412b-aa89-8e1bc502fc84-kube-api-access-wdbf7\" (UniqueName: \"kubernetes.io/projected/0ff84017-6a02-412b-aa89-8e1bc502fc84-kube-api-access-wdbf7\") pod \"0ff84017-6a02-412b-aa89-8e1bc502fc84\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " Nov 03 22:52:28.367745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.366786 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff84017-6a02-412b-aa89-8e1bc502fc84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff84017-6a02-412b-aa89-8e1bc502fc84-plugins-config-volume\") pod \"0ff84017-6a02-412b-aa89-8e1bc502fc84\" (UID: \"0ff84017-6a02-412b-aa89-8e1bc502fc84\") " Nov 03 22:52:28.367745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.367093 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff84017-6a02-412b-aa89-8e1bc502fc84-plugins-config-volume" pod "0ff84017-6a02-412b-aa89-8e1bc502fc84" (UID: "0ff84017-6a02-412b-aa89-8e1bc502fc84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:28.367745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.367710 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff84017-6a02-412b-aa89-8e1bc502fc84-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.372969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.372942 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff84017-6a02-412b-aa89-8e1bc502fc84-kube-api-access-wdbf7" pod "0ff84017-6a02-412b-aa89-8e1bc502fc84" (UID: "0ff84017-6a02-412b-aa89-8e1bc502fc84"). InnerVolumeSpecName "kube-api-access-wdbf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:28.403371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.403342 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:52:28.468693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.468612 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641a4a4c-cce4-49e8-bedd-1977115c8325-kube-api-access-tvcsm\" (UniqueName: \"kubernetes.io/projected/641a4a4c-cce4-49e8-bedd-1977115c8325-kube-api-access-tvcsm\") pod \"641a4a4c-cce4-49e8-bedd-1977115c8325\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " Nov 03 22:52:28.468926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.468905 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641a4a4c-cce4-49e8-bedd-1977115c8325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641a4a4c-cce4-49e8-bedd-1977115c8325-plugins-config-volume\") pod \"641a4a4c-cce4-49e8-bedd-1977115c8325\" (UID: \"641a4a4c-cce4-49e8-bedd-1977115c8325\") " Nov 03 22:52:28.469345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.469322 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641a4a4c-cce4-49e8-bedd-1977115c8325-plugins-config-volume" pod "641a4a4c-cce4-49e8-bedd-1977115c8325" (UID: "641a4a4c-cce4-49e8-bedd-1977115c8325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:28.469516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.469501 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641a4a4c-cce4-49e8-bedd-1977115c8325-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.469575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.469520 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbf7\" (UniqueName: \"kubernetes.io/projected/0ff84017-6a02-412b-aa89-8e1bc502fc84-kube-api-access-wdbf7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:28.470749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.470731 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641a4a4c-cce4-49e8-bedd-1977115c8325-kube-api-access-tvcsm" pod "641a4a4c-cce4-49e8-bedd-1977115c8325" (UID: "641a4a4c-cce4-49e8-bedd-1977115c8325"). InnerVolumeSpecName "kube-api-access-tvcsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:28.571089 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:28.571044 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcsm\" (UniqueName: \"kubernetes.io/projected/641a4a4c-cce4-49e8-bedd-1977115c8325-kube-api-access-tvcsm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.358689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.358652 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk" Nov 03 22:52:29.360216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.360192 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" Nov 03 22:52:29.360355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.360242 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k" event={"ID":"641a4a4c-cce4-49e8-bedd-1977115c8325","Type":"ContainerDied","Data":"188ced8d4e526b741e0b4ddc0b4c3aae9b9b6c291ef7526fdb9b615e56841406"} Nov 03 22:52:29.360355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.360323 5272 scope.go:122] "RemoveContainer" containerID="2ebbf6c29beaa6275254068335e482a2cfeb0fcda857bfc130fa636943b50db3" Nov 03 22:52:29.513933 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.513897 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk"] Nov 03 22:52:29.560347 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.560309 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12450a06-5c4cc76cd6-c9zgk"] Nov 03 22:52:29.621723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.621642 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k"] Nov 03 22:52:29.627364 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:29.627324 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12450a06-99d9f9f9-5tt6k"] Nov 03 22:52:31.497236 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:31.497199 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff84017-6a02-412b-aa89-8e1bc502fc84" path="/var/lib/kubelet/pods/0ff84017-6a02-412b-aa89-8e1bc502fc84/volumes" Nov 03 22:52:31.497715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:31.497700 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641a4a4c-cce4-49e8-bedd-1977115c8325" path="/var/lib/kubelet/pods/641a4a4c-cce4-49e8-bedd-1977115c8325/volumes" Nov 03 22:52:33.047154 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:33.047119 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d"] Nov 03 22:52:33.092942 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:33.092903 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e54a46a2-jqc4d"] Nov 03 22:52:33.495017 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:33.494936 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05346c7e-1f40-4a9a-a077-f2bfcd29eece" path="/var/lib/kubelet/pods/05346c7e-1f40-4a9a-a077-f2bfcd29eece/volumes" Nov 03 22:52:36.047613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:36.047576 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2"] Nov 03 22:52:36.093963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:36.093928 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7e88602-wpcf2"] Nov 03 22:52:37.496952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:37.496902 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c3efad-74fe-49aa-8544-6d651cd87953" path="/var/lib/kubelet/pods/26c3efad-74fe-49aa-8544-6d651cd87953/volumes" Nov 03 22:52:42.569553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:42.569517 5272 scope.go:122] "RemoveContainer" containerID="8d64e11c95b6bc1ce220af156fcdc75c7d3928861e6a32f9db9dc8ad67329773" Nov 03 22:52:42.580670 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:42.580648 5272 scope.go:122] "RemoveContainer" containerID="7e05145d71278764fc4733b29ba41d6b0b89346d61d0ceee17175d6d2a125202" Nov 03 22:52:53.606918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:53.606876 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc"] Nov 03 22:52:53.607460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:53.607159 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" containerID="cri-o://edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6" gracePeriod=30 Nov 03 22:52:54.501876 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:52:54.501833 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:52:56.316607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:56.316559 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:56.317043 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:56.316636 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:57.613130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.613096 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn"] Nov 03 22:52:57.614124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614102 5272 cpu_manager.go:414] "containerMap: removing container" podUID="641a4a4c-cce4-49e8-bedd-1977115c8325" containerName="payload-pre-processing" Nov 03 22:52:57.614124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614125 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641a4a4c-cce4-49e8-bedd-1977115c8325" containerName="payload-pre-processing" Nov 03 22:52:57.614232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614138 5272 cpu_manager.go:414] "containerMap: removing container" podUID="6ddedda6-fe9e-412b-b5a4-74a46141a6b8" containerName="revoke-keys" Nov 03 22:52:57.614232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614143 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddedda6-fe9e-412b-b5a4-74a46141a6b8" containerName="revoke-keys" Nov 03 22:52:57.614232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614187 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0ff84017-6a02-412b-aa89-8e1bc502fc84" containerName="payload-processing" Nov 03 22:52:57.614232 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614192 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff84017-6a02-412b-aa89-8e1bc502fc84" containerName="payload-processing" Nov 03 22:52:57.614358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614309 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="641a4a4c-cce4-49e8-bedd-1977115c8325" containerName="payload-pre-processing" Nov 03 22:52:57.614358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614322 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ddedda6-fe9e-412b-b5a4-74a46141a6b8" containerName="revoke-keys" Nov 03 22:52:57.614358 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.614336 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff84017-6a02-412b-aa89-8e1bc502fc84" containerName="payload-processing" Nov 03 22:52:57.619941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.619919 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.671740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.671709 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn"] Nov 03 22:52:57.713340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.713300 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:57.713340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.713334 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:52:57.713613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.713458 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:52:57.715463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.715438 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:52:57.791491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.791449 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.791646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.791520 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wnf\" (UniqueName: \"kubernetes.io/projected/2551c891-97ad-4d84-87ce-5038bd9c2680-kube-api-access-79wnf\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.791708 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.791642 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.893097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.892997 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.893097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.893075 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wnf\" (UniqueName: \"kubernetes.io/projected/2551c891-97ad-4d84-87ce-5038bd9c2680-kube-api-access-79wnf\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.893305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.893147 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.893756 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.893727 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.895474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.895456 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:57.950618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:57.950581 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wnf\" (UniqueName: \"kubernetes.io/projected/2551c891-97ad-4d84-87ce-5038bd9c2680-kube-api-access-79wnf\") pod \"maas-api-revoke-sub-9e656460-52gqn\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:58.234789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.234694 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:52:58.623572 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.623535 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn"] Nov 03 22:52:58.627088 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:52:58.627058 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2551c891_97ad_4d84_87ce_5038bd9c2680.slice/crio-962172886bb0b5f783e05f2bad44b42799bb841e1e198356a8dc21e4ac77d464 WatchSource:0}: Error finding container 962172886bb0b5f783e05f2bad44b42799bb841e1e198356a8dc21e4ac77d464: Status 404 returned error can't find the container with id 962172886bb0b5f783e05f2bad44b42799bb841e1e198356a8dc21e4ac77d464 Nov 03 22:52:58.884076 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.884051 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:52:58.909400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.909365 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-kube-api-access-bdjrj\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-kube-api-access-bdjrj\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.909851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.909828 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-socket\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.909989 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-podinfo\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910128 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-socket" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.910286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910252 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-envoy\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910299 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-certs\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910344 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-credential-socket\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910508 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910458 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-data\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910560 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910510 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-token\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910560 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910541 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istiod-ca-cert\") pod \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\" (UID: \"66e1f824-3797-4a06-815e-bc5ef21b3f2b\") " Nov 03 22:52:58.910678 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910652 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-certs" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.910904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.910884 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-data" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.911040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.911013 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-credential-socket" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.911194 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.911172 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istiod-ca-cert" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:58.911405 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.911372 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.911405 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.911397 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.911546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.911412 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.911546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.911425 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.917613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.917581 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-envoy" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.917613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.917602 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-kube-api-access-bdjrj" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "kube-api-access-bdjrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:58.917832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.917778 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-token" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:58.917896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:58.917796 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-podinfo" pod "66e1f824-3797-4a06-815e-bc5ef21b3f2b" (UID: "66e1f824-3797-4a06-815e-bc5ef21b3f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:59.013012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.012975 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.013012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.013007 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.013012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.013020 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.013250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.013032 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e1f824-3797-4a06-815e-bc5ef21b3f2b-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.013250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.013048 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjrj\" (UniqueName: \"kubernetes.io/projected/66e1f824-3797-4a06-815e-bc5ef21b3f2b-kube-api-access-bdjrj\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:59.610152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.610120 5272 generic.go:356] "Generic (PLEG): container finished" podID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerID="edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6" exitCode=0 Nov 03 22:52:59.610355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.610177 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" Nov 03 22:52:59.610355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.610200 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" event={"ID":"66e1f824-3797-4a06-815e-bc5ef21b3f2b","Type":"ContainerDied","Data":"edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6"} Nov 03 22:52:59.610355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.610266 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc" event={"ID":"66e1f824-3797-4a06-815e-bc5ef21b3f2b","Type":"ContainerDied","Data":"019c3bc724f9e13b41a57093a7763a4799faf3b072606058e6c5862804efd839"} Nov 03 22:52:59.610355 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.610297 5272 scope.go:122] "RemoveContainer" containerID="edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6" Nov 03 22:52:59.613956 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.613928 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" event={"ID":"2551c891-97ad-4d84-87ce-5038bd9c2680","Type":"ContainerStarted","Data":"61e3d444a9d9a2083a88d752f991c275322348f9b2d2ff00c8a3d3d599daa9bb"} Nov 03 22:52:59.614158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.614131 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" event={"ID":"2551c891-97ad-4d84-87ce-5038bd9c2680","Type":"ContainerStarted","Data":"962172886bb0b5f783e05f2bad44b42799bb841e1e198356a8dc21e4ac77d464"} Nov 03 22:52:59.624079 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.624055 5272 scope.go:122] "RemoveContainer" containerID="edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6" Nov 03 22:52:59.624420 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:52:59.624391 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6\": container with ID starting with edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6 not found: ID does not exist" containerID="edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6" Nov 03 22:52:59.624463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.624417 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6"} err="failed to get container status \"edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6\": rpc error: code = NotFound desc = could not find container \"edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6\": container with ID starting with edec61b5ff71d63aafa8f8c9b9228a96ca531897b712bcca8fe576ffc2f4cbd6 not found: ID does not exist" Nov 03 22:52:59.817173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.817115 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" podStartSLOduration=2.817100413 podStartE2EDuration="2.817100413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:59.814753768 +0000 UTC m=+1760.437062076" watchObservedRunningTime="2026-11-03 22:52:59.817100413 +0000 UTC m=+1760.439408720" Nov 03 22:52:59.925003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.924968 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc"] Nov 03 22:52:59.970689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:52:59.970649 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-th2cc"] Nov 03 22:53:00.625509 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:00.625471 5272 generic.go:356] "Generic (PLEG): container finished" podID="2551c891-97ad-4d84-87ce-5038bd9c2680" containerID="61e3d444a9d9a2083a88d752f991c275322348f9b2d2ff00c8a3d3d599daa9bb" exitCode=0 Nov 03 22:53:00.626039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:00.625554 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" event={"ID":"2551c891-97ad-4d84-87ce-5038bd9c2680","Type":"ContainerDied","Data":"61e3d444a9d9a2083a88d752f991c275322348f9b2d2ff00c8a3d3d599daa9bb"} Nov 03 22:53:01.496346 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.496315 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" path="/var/lib/kubelet/pods/66e1f824-3797-4a06-815e-bc5ef21b3f2b/volumes" Nov 03 22:53:01.801863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.801837 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:53:01.859538 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.859497 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-xks-ca\") pod \"2551c891-97ad-4d84-87ce-5038bd9c2680\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " Nov 03 22:53:01.859734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.859606 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2551c891-97ad-4d84-87ce-5038bd9c2680-kube-api-access-79wnf\" (UniqueName: \"kubernetes.io/projected/2551c891-97ad-4d84-87ce-5038bd9c2680-kube-api-access-79wnf\") pod \"2551c891-97ad-4d84-87ce-5038bd9c2680\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " Nov 03 22:53:01.859734 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.859680 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-openshift-ca\") pod \"2551c891-97ad-4d84-87ce-5038bd9c2680\" (UID: \"2551c891-97ad-4d84-87ce-5038bd9c2680\") " Nov 03 22:53:01.860056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.860032 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-openshift-ca" pod "2551c891-97ad-4d84-87ce-5038bd9c2680" (UID: "2551c891-97ad-4d84-87ce-5038bd9c2680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:01.860463 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.860442 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:01.861646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.861626 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-xks-ca" pod "2551c891-97ad-4d84-87ce-5038bd9c2680" (UID: "2551c891-97ad-4d84-87ce-5038bd9c2680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:01.861764 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.861747 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2551c891-97ad-4d84-87ce-5038bd9c2680-kube-api-access-79wnf" pod "2551c891-97ad-4d84-87ce-5038bd9c2680" (UID: "2551c891-97ad-4d84-87ce-5038bd9c2680"). InnerVolumeSpecName "kube-api-access-79wnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:01.962179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.962140 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2551c891-97ad-4d84-87ce-5038bd9c2680-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:01.962179 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:01.962172 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wnf\" (UniqueName: \"kubernetes.io/projected/2551c891-97ad-4d84-87ce-5038bd9c2680-kube-api-access-79wnf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.642618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:02.642589 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" Nov 03 22:53:02.642851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:02.642630 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn" event={"ID":"2551c891-97ad-4d84-87ce-5038bd9c2680","Type":"ContainerDied","Data":"962172886bb0b5f783e05f2bad44b42799bb841e1e198356a8dc21e4ac77d464"} Nov 03 22:53:02.642851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:02.642684 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="962172886bb0b5f783e05f2bad44b42799bb841e1e198356a8dc21e4ac77d464" Nov 03 22:53:09.822174 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:09.811530 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:53:24.503777 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:53:24.503740 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:53:42.641323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:42.641284 5272 scope.go:122] "RemoveContainer" containerID="595f09f5175feb646db56beeae68280cad3a905488ffa2a959d2d688f214e029" Nov 03 22:53:42.651851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:42.651827 5272 scope.go:122] "RemoveContainer" containerID="8282643b04198b41f1f55fab28a6285b8195ec253272f7a6027e7f8c48789716" Nov 03 22:53:53.065058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.064791 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff"] Nov 03 22:53:53.066918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.066853 5272 cpu_manager.go:414] "containerMap: removing container" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" Nov 03 22:53:53.067123 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.067102 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" Nov 03 22:53:53.067260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.067247 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2551c891-97ad-4d84-87ce-5038bd9c2680" containerName="revoke-keys" Nov 03 22:53:53.067345 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.067335 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2551c891-97ad-4d84-87ce-5038bd9c2680" containerName="revoke-keys" Nov 03 22:53:53.067707 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.067680 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e1f824-3797-4a06-815e-bc5ef21b3f2b" containerName="istio-proxy" Nov 03 22:53:53.067880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.067867 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2551c891-97ad-4d84-87ce-5038bd9c2680" containerName="revoke-keys" Nov 03 22:53:53.089278 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.088184 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.138749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.138698 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b745074-da33-4a98-9474-a09fa0ca0704-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.139302 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.139278 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r664\" (UniqueName: \"kubernetes.io/projected/9b745074-da33-4a98-9474-a09fa0ca0704-kube-api-access-5r664\") pod \"payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.146104 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.146063 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff"] Nov 03 22:53:53.173264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.165102 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d6296b-dockercfg-c2hx5\"" Nov 03 22:53:53.173264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.165243 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d6296b\"" Nov 03 22:53:53.259641 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.256641 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r664\" (UniqueName: \"kubernetes.io/projected/9b745074-da33-4a98-9474-a09fa0ca0704-kube-api-access-5r664\") pod \"payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.259641 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.256717 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b745074-da33-4a98-9474-a09fa0ca0704-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.259641 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.257762 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b745074-da33-4a98-9474-a09fa0ca0704-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.371826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.362519 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r664\" (UniqueName: \"kubernetes.io/projected/9b745074-da33-4a98-9474-a09fa0ca0704-kube-api-access-5r664\") pod \"payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.371826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.365028 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7"] Nov 03 22:53:53.405468 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.405439 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.425791 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.425754 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7"] Nov 03 22:53:53.439940 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.439908 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:53.566615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.566578 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.566821 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.566781 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb2tc\" (UniqueName: \"kubernetes.io/projected/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-kube-api-access-gb2tc\") pod \"payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.668445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.668350 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb2tc\" (UniqueName: \"kubernetes.io/projected/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-kube-api-access-gb2tc\") pod \"payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.668638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.668509 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.669267 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.669241 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.726026 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.725992 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb2tc\" (UniqueName: \"kubernetes.io/projected/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-kube-api-access-gb2tc\") pod \"payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.728813 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.728784 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:53.874833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:53.872543 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff"] Nov 03 22:53:54.110212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.110171 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7"] Nov 03 22:53:54.115860 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:53:54.115822 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b491f0_73d8_4bb9_90ae_1ab6b489b8a9.slice/crio-348504dcbaf4af5f40678bc436cb56946653ef7c3cc1e719c64022369bfdedbb WatchSource:0}: Error finding container 348504dcbaf4af5f40678bc436cb56946653ef7c3cc1e719c64022369bfdedbb: Status 404 returned error can't find the container with id 348504dcbaf4af5f40678bc436cb56946653ef7c3cc1e719c64022369bfdedbb Nov 03 22:53:54.445876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.445835 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" event={"ID":"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9","Type":"ContainerStarted","Data":"d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1"} Nov 03 22:53:54.446071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.445903 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:53:54.446071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.445925 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" event={"ID":"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9","Type":"ContainerStarted","Data":"348504dcbaf4af5f40678bc436cb56946653ef7c3cc1e719c64022369bfdedbb"} Nov 03 22:53:54.447595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.447563 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" event={"ID":"9b745074-da33-4a98-9474-a09fa0ca0704","Type":"ContainerStarted","Data":"70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71"} Nov 03 22:53:54.447721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.447615 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" event={"ID":"9b745074-da33-4a98-9474-a09fa0ca0704","Type":"ContainerStarted","Data":"768625963f3f00b9d023947384dc40d7a120e05f9fd1f7da950bfc30dfe7b4dc"} Nov 03 22:53:54.447721 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.447669 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:53:54.502983 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:53:54.502949 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:53:54.560097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.560043 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" podStartSLOduration=1.560028476 podStartE2EDuration="1.560028476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:54.558109837 +0000 UTC m=+1815.180418144" watchObservedRunningTime="2026-11-03 22:53:54.560028476 +0000 UTC m=+1815.182336784" Nov 03 22:53:54.686828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:54.671553 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" podStartSLOduration=1.671537029 podStartE2EDuration="1.671537029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:54.666154098 +0000 UTC m=+1815.288462407" watchObservedRunningTime="2026-11-03 22:53:54.671537029 +0000 UTC m=+1815.293845338" Nov 03 22:53:55.557591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:53:55.557559 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:54:05.458952 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:05.458917 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 22:54:05.459422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:05.458990 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 22:54:15.047208 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:15.047163 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5"] Nov 03 22:54:15.093510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:15.093482 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20a8c1c7-sgxt5"] Nov 03 22:54:15.494716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:15.494627 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b844aaa-b202-44c2-acc8-3e004029bca0" path="/var/lib/kubelet/pods/8b844aaa-b202-44c2-acc8-3e004029bca0/volumes" Nov 03 22:54:24.502006 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:54:24.501970 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:54:24.690130 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:24.690096 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl"] Nov 03 22:54:24.690371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:24.690345 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" containerID="cri-o://bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c" gracePeriod=30 Nov 03 22:54:25.896529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:25.896495 5272 patch_prober.go:29] interesting pod/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:54:25.896925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:25.896549 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:54:28.993490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:28.993453 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k"] Nov 03 22:54:28.998777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:28.998756 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.053318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.053289 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k"] Nov 03 22:54:29.078609 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.078578 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.078968 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.078937 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.079085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.078993 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tx7\" (UniqueName: \"kubernetes.io/projected/2327346f-cdc8-45fa-b48e-680d5345640a-kube-api-access-w6tx7\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.094401 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.094376 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:54:29.094401 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.094395 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:29.094549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.094385 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:54:29.094549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.094425 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:54:29.180235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.180209 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.180372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.180248 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tx7\" (UniqueName: \"kubernetes.io/projected/2327346f-cdc8-45fa-b48e-680d5345640a-kube-api-access-w6tx7\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.180372 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.180313 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.181069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.181049 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.182752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.182732 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.239271 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.239250 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tx7\" (UniqueName: \"kubernetes.io/projected/2327346f-cdc8-45fa-b48e-680d5345640a-kube-api-access-w6tx7\") pod \"maas-api-revoke-keys-e2e-api-a-922795-r6r4k\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.314223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.314198 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:29.691251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.691171 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k"] Nov 03 22:54:29.695937 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:54:29.695902 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2327346f_cdc8_45fa_b48e_680d5345640a.slice/crio-c5862d9c4ae9cc633fea4129ca2bfb33df78f24af58897fdcc0887470f7eca2b WatchSource:0}: Error finding container c5862d9c4ae9cc633fea4129ca2bfb33df78f24af58897fdcc0887470f7eca2b: Status 404 returned error can't find the container with id c5862d9c4ae9cc633fea4129ca2bfb33df78f24af58897fdcc0887470f7eca2b Nov 03 22:54:29.776879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.776841 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" event={"ID":"2327346f-cdc8-45fa-b48e-680d5345640a","Type":"ContainerStarted","Data":"c5862d9c4ae9cc633fea4129ca2bfb33df78f24af58897fdcc0887470f7eca2b"} Nov 03 22:54:29.969329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.969305 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:54:29.996213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996190 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-kube-api-access-49579\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-kube-api-access-49579\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996228 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-token\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996264 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-credential-socket\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996316 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-data\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996436 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-podinfo\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996499 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-socket\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996592 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istiod-ca-cert\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996578 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-credential-socket" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.996919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996645 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-data" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.996919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996703 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-certs\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996750 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-envoy\") pod \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\" (UID: \"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf\") " Nov 03 22:54:29.996919 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996772 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-socket" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.997265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996963 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-certs" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.997265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.996982 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istiod-ca-cert" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:29.997474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.997448 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.997474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.997475 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.997669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.997489 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.997669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.997502 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.997669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.997516 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.998857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.998785 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-podinfo" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:29.998857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.998818 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-kube-api-access-49579" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "kube-api-access-49579". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:29.998984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.998924 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-token" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:29.999068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:29.999049 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-envoy" pod "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" (UID: "2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:30.098626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.098601 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49579\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-kube-api-access-49579\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.098626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.098628 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.098767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.098637 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.098767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.098646 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:30.785849 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.785796 5272 generic.go:356] "Generic (PLEG): container finished" podID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerID="bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c" exitCode=0 Nov 03 22:54:30.785991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.785851 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" event={"ID":"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf","Type":"ContainerDied","Data":"bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c"} Nov 03 22:54:30.785991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.785878 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" Nov 03 22:54:30.785991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.785916 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl" event={"ID":"2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf","Type":"ContainerDied","Data":"865bc2100a99affa8a35b1f7d385d6e1c808256f0e2b23a874821e0198de4f35"} Nov 03 22:54:30.785991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.785940 5272 scope.go:122] "RemoveContainer" containerID="bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c" Nov 03 22:54:30.787697 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.787673 5272 generic.go:356] "Generic (PLEG): container finished" podID="2327346f-cdc8-45fa-b48e-680d5345640a" containerID="695a8d18361b511b2585fd2fee4157455d60d7796cc73c20b6ea029eadefd7d6" exitCode=0 Nov 03 22:54:30.787846 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.787825 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" event={"ID":"2327346f-cdc8-45fa-b48e-680d5345640a","Type":"ContainerDied","Data":"695a8d18361b511b2585fd2fee4157455d60d7796cc73c20b6ea029eadefd7d6"} Nov 03 22:54:30.797399 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.797377 5272 scope.go:122] "RemoveContainer" containerID="bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c" Nov 03 22:54:30.797646 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:54:30.797629 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c\": container with ID starting with bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c not found: ID does not exist" containerID="bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c" Nov 03 22:54:30.797696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:30.797653 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c"} err="failed to get container status \"bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c\": rpc error: code = NotFound desc = could not find container \"bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c\": container with ID starting with bf00c0a383467aabf6a30e8aadb871c11506227b8bf2118bc9dbabf588eb685c not found: ID does not exist" Nov 03 22:54:31.050743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:31.050706 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl"] Nov 03 22:54:31.054524 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:31.054497 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-nnsbl"] Nov 03 22:54:31.495968 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:31.495896 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" path="/var/lib/kubelet/pods/2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf/volumes" Nov 03 22:54:32.001320 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.001292 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:32.035277 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.035254 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-openshift-ca\") pod \"2327346f-cdc8-45fa-b48e-680d5345640a\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " Nov 03 22:54:32.035398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.035339 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2327346f-cdc8-45fa-b48e-680d5345640a-kube-api-access-w6tx7\" (UniqueName: \"kubernetes.io/projected/2327346f-cdc8-45fa-b48e-680d5345640a-kube-api-access-w6tx7\") pod \"2327346f-cdc8-45fa-b48e-680d5345640a\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " Nov 03 22:54:32.035398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.035370 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-xks-ca\") pod \"2327346f-cdc8-45fa-b48e-680d5345640a\" (UID: \"2327346f-cdc8-45fa-b48e-680d5345640a\") " Nov 03 22:54:32.035626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.035601 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-openshift-ca" pod "2327346f-cdc8-45fa-b48e-680d5345640a" (UID: "2327346f-cdc8-45fa-b48e-680d5345640a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:32.035943 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.035922 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:32.037351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.037325 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-xks-ca" pod "2327346f-cdc8-45fa-b48e-680d5345640a" (UID: "2327346f-cdc8-45fa-b48e-680d5345640a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:32.037440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.037404 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2327346f-cdc8-45fa-b48e-680d5345640a-kube-api-access-w6tx7" pod "2327346f-cdc8-45fa-b48e-680d5345640a" (UID: "2327346f-cdc8-45fa-b48e-680d5345640a"). InnerVolumeSpecName "kube-api-access-w6tx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:32.137053 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.137024 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tx7\" (UniqueName: \"kubernetes.io/projected/2327346f-cdc8-45fa-b48e-680d5345640a-kube-api-access-w6tx7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:32.137053 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.137050 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327346f-cdc8-45fa-b48e-680d5345640a-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:32.805035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.805005 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" Nov 03 22:54:32.805035 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.805023 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k" event={"ID":"2327346f-cdc8-45fa-b48e-680d5345640a","Type":"ContainerDied","Data":"c5862d9c4ae9cc633fea4129ca2bfb33df78f24af58897fdcc0887470f7eca2b"} Nov 03 22:54:32.805239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:32.805069 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5862d9c4ae9cc633fea4129ca2bfb33df78f24af58897fdcc0887470f7eca2b" Nov 03 22:54:36.714342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.714309 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c"] Nov 03 22:54:36.715190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.715169 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" Nov 03 22:54:36.715190 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.715189 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" Nov 03 22:54:36.715316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.715198 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2327346f-cdc8-45fa-b48e-680d5345640a" containerName="revoke-keys" Nov 03 22:54:36.715316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.715203 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2327346f-cdc8-45fa-b48e-680d5345640a" containerName="revoke-keys" Nov 03 22:54:36.715384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.715354 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecd2d08-ed0c-401e-a875-a9d7bf75b4cf" containerName="istio-proxy" Nov 03 22:54:36.715384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.715364 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2327346f-cdc8-45fa-b48e-680d5345640a" containerName="revoke-keys" Nov 03 22:54:36.719185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.719168 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.773202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.773173 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c"] Nov 03 22:54:36.790085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.790057 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgzb\" (UniqueName: \"kubernetes.io/projected/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-kube-api-access-pxgzb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.790257 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.790183 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.790392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.790376 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.814978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.814946 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:54:36.814978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.814955 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:54:36.814978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.814955 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:54:36.815231 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.814956 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:36.891455 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.891414 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.891712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.891693 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.891778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.891750 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgzb\" (UniqueName: \"kubernetes.io/projected/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-kube-api-access-pxgzb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.892145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.892124 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.894117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.894094 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:36.972591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:36.972483 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgzb\" (UniqueName: \"kubernetes.io/projected/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-kube-api-access-pxgzb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:37.032864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:37.032830 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:37.415652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:37.415619 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c"] Nov 03 22:54:37.421022 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:54:37.420995 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7fb5aa_8ef1_4c36_8bb3_14b59adf4597.slice/crio-cccaa6e8ac5d48e3fba43a87a1a8332a0c505b922bbc5115b38ae8c6ab853c62 WatchSource:0}: Error finding container cccaa6e8ac5d48e3fba43a87a1a8332a0c505b922bbc5115b38ae8c6ab853c62: Status 404 returned error can't find the container with id cccaa6e8ac5d48e3fba43a87a1a8332a0c505b922bbc5115b38ae8c6ab853c62 Nov 03 22:54:37.845482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:37.845446 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" event={"ID":"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597","Type":"ContainerStarted","Data":"69bccd22ebcfbec10f285a677b8cc5e6d21eccb9cc4ee191b73f7a0d19806294"} Nov 03 22:54:37.845894 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:37.845505 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" event={"ID":"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597","Type":"ContainerStarted","Data":"cccaa6e8ac5d48e3fba43a87a1a8332a0c505b922bbc5115b38ae8c6ab853c62"} Nov 03 22:54:37.957984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:37.957930 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" podStartSLOduration=1.957916565 podStartE2EDuration="1.957916565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:37.954392491 +0000 UTC m=+1858.576700801" watchObservedRunningTime="2026-11-03 22:54:37.957916565 +0000 UTC m=+1858.580224903" Nov 03 22:54:38.854276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:38.854239 5272 generic.go:356] "Generic (PLEG): container finished" podID="ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" containerID="69bccd22ebcfbec10f285a677b8cc5e6d21eccb9cc4ee191b73f7a0d19806294" exitCode=0 Nov 03 22:54:38.854657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:38.854293 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" event={"ID":"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597","Type":"ContainerDied","Data":"69bccd22ebcfbec10f285a677b8cc5e6d21eccb9cc4ee191b73f7a0d19806294"} Nov 03 22:54:40.030142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.030118 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:40.152292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.152194 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-openshift-ca\") pod \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " Nov 03 22:54:40.152454 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.152329 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-xks-ca\") pod \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " Nov 03 22:54:40.152527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.152449 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-kube-api-access-pxgzb\" (UniqueName: \"kubernetes.io/projected/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-kube-api-access-pxgzb\") pod \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\" (UID: \"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597\") " Nov 03 22:54:40.152639 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.152614 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-openshift-ca" pod "ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" (UID: "ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:40.153360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.153330 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:40.159865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.159836 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-xks-ca" pod "ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" (UID: "ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:40.159997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.159970 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-kube-api-access-pxgzb" pod "ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" (UID: "ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597"). InnerVolumeSpecName "kube-api-access-pxgzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:40.255690 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.255652 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxgzb\" (UniqueName: \"kubernetes.io/projected/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-kube-api-access-pxgzb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:40.255690 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.255688 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:40.869548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.869519 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" Nov 03 22:54:40.869749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.869583 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c" event={"ID":"ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597","Type":"ContainerDied","Data":"cccaa6e8ac5d48e3fba43a87a1a8332a0c505b922bbc5115b38ae8c6ab853c62"} Nov 03 22:54:40.869749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:40.869639 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cccaa6e8ac5d48e3fba43a87a1a8332a0c505b922bbc5115b38ae8c6ab853c62" Nov 03 22:54:42.692761 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:42.692680 5272 scope.go:122] "RemoveContainer" containerID="33f3e4b3de5bdb7717f87646d147c66c4ec16e32b58aabd81f50a3eea6f6045e" Nov 03 22:54:43.533053 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:43.533025 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42"] Nov 03 22:54:43.614938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:43.614904 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq"] Nov 03 22:54:43.664574 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:43.664540 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2"] Nov 03 22:54:43.692563 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:43.692529 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-hkkr2"] Nov 03 22:54:45.500739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:45.500688 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ddedda6-fe9e-412b-b5a4-74a46141a6b8" path="/var/lib/kubelet/pods/6ddedda6-fe9e-412b-b5a4-74a46141a6b8/volumes" Nov 03 22:54:48.533486 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.533435 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" podUID="d3927a87-6644-42c5-a642-e7ec788f7e43" containerName="payload-processing" containerID="cri-o://9e051dee4ff804fc5e2237d88f466b7ad6a7abe5a3a7ba2bfb47c83b5b412183" gracePeriod=25 Nov 03 22:54:48.615379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.615347 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" podUID="35d3ddf0-3385-463d-9c06-1c79fd8faa57" containerName="payload-pre-processing" containerID="cri-o://1384a42c43fb1fa3b90b701fd6f9a824e3c89277c4adfa941c7151ebb9d12b63" gracePeriod=25 Nov 03 22:54:48.964216 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.964185 5272 generic.go:356] "Generic (PLEG): container finished" podID="d3927a87-6644-42c5-a642-e7ec788f7e43" containerID="9e051dee4ff804fc5e2237d88f466b7ad6a7abe5a3a7ba2bfb47c83b5b412183" exitCode=0 Nov 03 22:54:48.964488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.964253 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" event={"ID":"d3927a87-6644-42c5-a642-e7ec788f7e43","Type":"ContainerDied","Data":"9e051dee4ff804fc5e2237d88f466b7ad6a7abe5a3a7ba2bfb47c83b5b412183"} Nov 03 22:54:48.964488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.964314 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" event={"ID":"d3927a87-6644-42c5-a642-e7ec788f7e43","Type":"ContainerDied","Data":"7a848928b2bbf3f84c03820c35c943407586a02a0500afd06e0c9d02bc77ba09"} Nov 03 22:54:48.964488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.964346 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a848928b2bbf3f84c03820c35c943407586a02a0500afd06e0c9d02bc77ba09" Nov 03 22:54:48.964673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.964597 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:54:48.965913 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.965893 5272 generic.go:356] "Generic (PLEG): container finished" podID="35d3ddf0-3385-463d-9c06-1c79fd8faa57" containerID="1384a42c43fb1fa3b90b701fd6f9a824e3c89277c4adfa941c7151ebb9d12b63" exitCode=0 Nov 03 22:54:48.966018 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:48.965973 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" event={"ID":"35d3ddf0-3385-463d-9c06-1c79fd8faa57","Type":"ContainerDied","Data":"1384a42c43fb1fa3b90b701fd6f9a824e3c89277c4adfa941c7151ebb9d12b63"} Nov 03 22:54:49.000723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.000704 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:54:49.053607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.053586 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3927a87-6644-42c5-a642-e7ec788f7e43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3927a87-6644-42c5-a642-e7ec788f7e43-plugins-config-volume\") pod \"d3927a87-6644-42c5-a642-e7ec788f7e43\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " Nov 03 22:54:49.053703 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.053676 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3927a87-6644-42c5-a642-e7ec788f7e43-kube-api-access-6jmtw\" (UniqueName: \"kubernetes.io/projected/d3927a87-6644-42c5-a642-e7ec788f7e43-kube-api-access-6jmtw\") pod \"d3927a87-6644-42c5-a642-e7ec788f7e43\" (UID: \"d3927a87-6644-42c5-a642-e7ec788f7e43\") " Nov 03 22:54:49.053781 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.053770 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d3ddf0-3385-463d-9c06-1c79fd8faa57-kube-api-access-g26kw\" (UniqueName: \"kubernetes.io/projected/35d3ddf0-3385-463d-9c06-1c79fd8faa57-kube-api-access-g26kw\") pod \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " Nov 03 22:54:49.053862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.053835 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35d3ddf0-3385-463d-9c06-1c79fd8faa57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d3ddf0-3385-463d-9c06-1c79fd8faa57-plugins-config-volume\") pod \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\" (UID: \"35d3ddf0-3385-463d-9c06-1c79fd8faa57\") " Nov 03 22:54:49.053984 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.053959 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3927a87-6644-42c5-a642-e7ec788f7e43-plugins-config-volume" pod "d3927a87-6644-42c5-a642-e7ec788f7e43" (UID: "d3927a87-6644-42c5-a642-e7ec788f7e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:49.054261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.054241 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d3ddf0-3385-463d-9c06-1c79fd8faa57-plugins-config-volume" pod "35d3ddf0-3385-463d-9c06-1c79fd8faa57" (UID: "35d3ddf0-3385-463d-9c06-1c79fd8faa57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:49.054488 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.054473 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d3ddf0-3385-463d-9c06-1c79fd8faa57-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:49.054527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.054492 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3927a87-6644-42c5-a642-e7ec788f7e43-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:49.055902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.055878 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d3ddf0-3385-463d-9c06-1c79fd8faa57-kube-api-access-g26kw" pod "35d3ddf0-3385-463d-9c06-1c79fd8faa57" (UID: "35d3ddf0-3385-463d-9c06-1c79fd8faa57"). InnerVolumeSpecName "kube-api-access-g26kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:49.055961 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.055908 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3927a87-6644-42c5-a642-e7ec788f7e43-kube-api-access-6jmtw" pod "d3927a87-6644-42c5-a642-e7ec788f7e43" (UID: "d3927a87-6644-42c5-a642-e7ec788f7e43"). InnerVolumeSpecName "kube-api-access-6jmtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:49.155420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.155355 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmtw\" (UniqueName: \"kubernetes.io/projected/d3927a87-6644-42c5-a642-e7ec788f7e43-kube-api-access-6jmtw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:49.155420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.155380 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26kw\" (UniqueName: \"kubernetes.io/projected/35d3ddf0-3385-463d-9c06-1c79fd8faa57-kube-api-access-g26kw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:49.974903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.974863 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" event={"ID":"35d3ddf0-3385-463d-9c06-1c79fd8faa57","Type":"ContainerDied","Data":"febe8efee8f7ec74e90c9cf76feeac39a845099ad18e79c10bca6d6831149dc4"} Nov 03 22:54:49.975341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.974938 5272 scope.go:122] "RemoveContainer" containerID="1384a42c43fb1fa3b90b701fd6f9a824e3c89277c4adfa941c7151ebb9d12b63" Nov 03 22:54:49.975341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.974880 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42" Nov 03 22:54:49.975341 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:49.974880 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq" Nov 03 22:54:50.183328 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:50.183299 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42"] Nov 03 22:54:50.225743 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:50.225652 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922795-56d4c4f7db-hjv42"] Nov 03 22:54:50.332224 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:50.332197 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq"] Nov 03 22:54:50.336979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:50.336949 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922795-559bb85d47-l8qpq"] Nov 03 22:54:50.761618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:50.761586 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229"] Nov 03 22:54:50.880091 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:50.880059 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr"] Nov 03 22:54:51.464866 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.464830 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98"] Nov 03 22:54:51.465711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465689 5272 cpu_manager.go:414] "containerMap: removing container" podUID="35d3ddf0-3385-463d-9c06-1c79fd8faa57" containerName="payload-pre-processing" Nov 03 22:54:51.465711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465708 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d3ddf0-3385-463d-9c06-1c79fd8faa57" containerName="payload-pre-processing" Nov 03 22:54:51.465921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465719 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d3927a87-6644-42c5-a642-e7ec788f7e43" containerName="payload-processing" Nov 03 22:54:51.465921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465724 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3927a87-6644-42c5-a642-e7ec788f7e43" containerName="payload-processing" Nov 03 22:54:51.465921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465758 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" containerName="revoke-keys" Nov 03 22:54:51.465921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465763 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" containerName="revoke-keys" Nov 03 22:54:51.465921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465880 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d3ddf0-3385-463d-9c06-1c79fd8faa57" containerName="payload-pre-processing" Nov 03 22:54:51.465921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465887 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3927a87-6644-42c5-a642-e7ec788f7e43" containerName="payload-processing" Nov 03 22:54:51.465921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.465899 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" containerName="revoke-keys" Nov 03 22:54:51.471116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.471100 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.495447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.495423 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35d3ddf0-3385-463d-9c06-1c79fd8faa57" path="/var/lib/kubelet/pods/35d3ddf0-3385-463d-9c06-1c79fd8faa57/volumes" Nov 03 22:54:51.495904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.495889 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3927a87-6644-42c5-a642-e7ec788f7e43" path="/var/lib/kubelet/pods/d3927a87-6644-42c5-a642-e7ec788f7e43/volumes" Nov 03 22:54:51.520433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.520411 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:54:51.520552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.520517 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:54:51.520921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.520900 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:54:51.521097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.521079 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:51.524160 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.524139 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98"] Nov 03 22:54:51.600469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.600442 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.600626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.600537 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.600626 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.600572 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46mn\" (UniqueName: \"kubernetes.io/projected/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-kube-api-access-q46mn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.701613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.701587 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.701731 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.701646 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.701731 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.701665 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46mn\" (UniqueName: \"kubernetes.io/projected/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-kube-api-access-q46mn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.702253 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.702234 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.703930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.703911 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.758869 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.758819 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46mn\" (UniqueName: \"kubernetes.io/projected/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-kube-api-access-q46mn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:51.785532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:51.785512 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:52.163398 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:52.163361 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98"] Nov 03 22:54:52.168871 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:54:52.168842 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca06d0f3_bdeb_4d9a_8d71_479d949be05e.slice/crio-8e323c6ac48ed106ba16c511380e91766fe24d43fc277b8db9e638bdb3bd30c9 WatchSource:0}: Error finding container 8e323c6ac48ed106ba16c511380e91766fe24d43fc277b8db9e638bdb3bd30c9: Status 404 returned error can't find the container with id 8e323c6ac48ed106ba16c511380e91766fe24d43fc277b8db9e638bdb3bd30c9 Nov 03 22:54:53.024030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:53.023954 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" event={"ID":"ca06d0f3-bdeb-4d9a-8d71-479d949be05e","Type":"ContainerStarted","Data":"d09bc117f4dc13afba4c36604cc683499fd4225d5ffe12b0565ea36845c2d9fd"} Nov 03 22:54:53.024030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:53.024014 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" event={"ID":"ca06d0f3-bdeb-4d9a-8d71-479d949be05e","Type":"ContainerStarted","Data":"8e323c6ac48ed106ba16c511380e91766fe24d43fc277b8db9e638bdb3bd30c9"} Nov 03 22:54:53.137077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:53.137025 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" podStartSLOduration=2.137011348 podStartE2EDuration="2.137011348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:53.133456711 +0000 UTC m=+1873.755765018" watchObservedRunningTime="2026-11-03 22:54:53.137011348 +0000 UTC m=+1873.759319655" Nov 03 22:54:54.045684 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:54.045647 5272 generic.go:356] "Generic (PLEG): container finished" podID="ca06d0f3-bdeb-4d9a-8d71-479d949be05e" containerID="d09bc117f4dc13afba4c36604cc683499fd4225d5ffe12b0565ea36845c2d9fd" exitCode=0 Nov 03 22:54:54.046248 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:54.045741 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" event={"ID":"ca06d0f3-bdeb-4d9a-8d71-479d949be05e","Type":"ContainerDied","Data":"d09bc117f4dc13afba4c36604cc683499fd4225d5ffe12b0565ea36845c2d9fd"} Nov 03 22:54:54.502045 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:54:54.501961 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:54:55.205421 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.205398 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:55.271499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.271469 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-openshift-ca\") pod \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " Nov 03 22:54:55.271653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.271563 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-xks-ca\") pod \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " Nov 03 22:54:55.271782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.271652 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-kube-api-access-q46mn\" (UniqueName: \"kubernetes.io/projected/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-kube-api-access-q46mn\") pod \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\" (UID: \"ca06d0f3-bdeb-4d9a-8d71-479d949be05e\") " Nov 03 22:54:55.271875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.271845 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-openshift-ca" pod "ca06d0f3-bdeb-4d9a-8d71-479d949be05e" (UID: "ca06d0f3-bdeb-4d9a-8d71-479d949be05e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:55.272236 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.272216 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:55.273661 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.273640 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-xks-ca" pod "ca06d0f3-bdeb-4d9a-8d71-479d949be05e" (UID: "ca06d0f3-bdeb-4d9a-8d71-479d949be05e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:55.273880 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.273861 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-kube-api-access-q46mn" pod "ca06d0f3-bdeb-4d9a-8d71-479d949be05e" (UID: "ca06d0f3-bdeb-4d9a-8d71-479d949be05e"). InnerVolumeSpecName "kube-api-access-q46mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:55.373397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.373328 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:55.373397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.373356 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46mn\" (UniqueName: \"kubernetes.io/projected/ca06d0f3-bdeb-4d9a-8d71-479d949be05e-kube-api-access-q46mn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:55.881045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:55.881010 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" podUID="ee77aaff-2035-49ba-9d77-5f13a2ad4a30" containerName="payload-pre-processing" containerID="cri-o://9f82c580ab7dceabb80966a7e19fc75db03b6477836fa883d2e4420ae02d5432" gracePeriod=25 Nov 03 22:54:56.007876 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.007842 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" podUID="2f5ffd82-72be-42b9-be1b-b7ac56057509" containerName="payload-processing" containerID="cri-o://c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3" gracePeriod=25 Nov 03 22:54:56.071389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.071360 5272 generic.go:356] "Generic (PLEG): container finished" podID="ee77aaff-2035-49ba-9d77-5f13a2ad4a30" containerID="9f82c580ab7dceabb80966a7e19fc75db03b6477836fa883d2e4420ae02d5432" exitCode=0 Nov 03 22:54:56.071562 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.071546 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" event={"ID":"ee77aaff-2035-49ba-9d77-5f13a2ad4a30","Type":"ContainerDied","Data":"9f82c580ab7dceabb80966a7e19fc75db03b6477836fa883d2e4420ae02d5432"} Nov 03 22:54:56.073235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.073205 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" event={"ID":"ca06d0f3-bdeb-4d9a-8d71-479d949be05e","Type":"ContainerDied","Data":"8e323c6ac48ed106ba16c511380e91766fe24d43fc277b8db9e638bdb3bd30c9"} Nov 03 22:54:56.073344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.073261 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e323c6ac48ed106ba16c511380e91766fe24d43fc277b8db9e638bdb3bd30c9" Nov 03 22:54:56.073344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.073215 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98" Nov 03 22:54:56.123228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.123205 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:54:56.202842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.190143 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-plugins-config-volume\") pod \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " Nov 03 22:54:56.202842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.190447 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-kube-api-access-wjsf5\" (UniqueName: \"kubernetes.io/projected/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-kube-api-access-wjsf5\") pod \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\" (UID: \"ee77aaff-2035-49ba-9d77-5f13a2ad4a30\") " Nov 03 22:54:56.202842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.192031 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-plugins-config-volume" pod "ee77aaff-2035-49ba-9d77-5f13a2ad4a30" (UID: "ee77aaff-2035-49ba-9d77-5f13a2ad4a30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:56.202842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.193830 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-kube-api-access-wjsf5" pod "ee77aaff-2035-49ba-9d77-5f13a2ad4a30" (UID: "ee77aaff-2035-49ba-9d77-5f13a2ad4a30"). InnerVolumeSpecName "kube-api-access-wjsf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:56.292742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.292717 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjsf5\" (UniqueName: \"kubernetes.io/projected/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-kube-api-access-wjsf5\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:56.292742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.292742 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee77aaff-2035-49ba-9d77-5f13a2ad4a30-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:56.304833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.304781 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:54:56.393162 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.393140 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5ffd82-72be-42b9-be1b-b7ac56057509-kube-api-access-v57hf\" (UniqueName: \"kubernetes.io/projected/2f5ffd82-72be-42b9-be1b-b7ac56057509-kube-api-access-v57hf\") pod \"2f5ffd82-72be-42b9-be1b-b7ac56057509\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " Nov 03 22:54:56.393360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.393347 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5ffd82-72be-42b9-be1b-b7ac56057509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5ffd82-72be-42b9-be1b-b7ac56057509-plugins-config-volume\") pod \"2f5ffd82-72be-42b9-be1b-b7ac56057509\" (UID: \"2f5ffd82-72be-42b9-be1b-b7ac56057509\") " Nov 03 22:54:56.393709 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.393680 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5ffd82-72be-42b9-be1b-b7ac56057509-plugins-config-volume" pod "2f5ffd82-72be-42b9-be1b-b7ac56057509" (UID: "2f5ffd82-72be-42b9-be1b-b7ac56057509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:56.395136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.395112 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5ffd82-72be-42b9-be1b-b7ac56057509-kube-api-access-v57hf" pod "2f5ffd82-72be-42b9-be1b-b7ac56057509" (UID: "2f5ffd82-72be-42b9-be1b-b7ac56057509"). InnerVolumeSpecName "kube-api-access-v57hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:56.495505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.495432 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5ffd82-72be-42b9-be1b-b7ac56057509-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:56.495505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:56.495462 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57hf\" (UniqueName: \"kubernetes.io/projected/2f5ffd82-72be-42b9-be1b-b7ac56057509-kube-api-access-v57hf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:57.081890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.081856 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" Nov 03 22:54:57.082088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.081860 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr" event={"ID":"ee77aaff-2035-49ba-9d77-5f13a2ad4a30","Type":"ContainerDied","Data":"00b27b5f6f04c66b8a9577033d2143db2434bcc8d1fbb48546e52cc379c06599"} Nov 03 22:54:57.082088 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.082009 5272 scope.go:122] "RemoveContainer" containerID="9f82c580ab7dceabb80966a7e19fc75db03b6477836fa883d2e4420ae02d5432" Nov 03 22:54:57.083322 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.083301 5272 generic.go:356] "Generic (PLEG): container finished" podID="2f5ffd82-72be-42b9-be1b-b7ac56057509" containerID="c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3" exitCode=0 Nov 03 22:54:57.083441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.083327 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" event={"ID":"2f5ffd82-72be-42b9-be1b-b7ac56057509","Type":"ContainerDied","Data":"c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3"} Nov 03 22:54:57.083441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.083359 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" Nov 03 22:54:57.083441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.083370 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229" event={"ID":"2f5ffd82-72be-42b9-be1b-b7ac56057509","Type":"ContainerDied","Data":"39c2990b872d724d08f1226f6a366243504868fb809a970da5c7a1e9833ab29f"} Nov 03 22:54:57.094620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.094603 5272 scope.go:122] "RemoveContainer" containerID="c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3" Nov 03 22:54:57.103740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.103722 5272 scope.go:122] "RemoveContainer" containerID="c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3" Nov 03 22:54:57.104044 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:54:57.104022 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3\": container with ID starting with c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3 not found: ID does not exist" containerID="c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3" Nov 03 22:54:57.104102 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.104051 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3"} err="failed to get container status \"c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3\": rpc error: code = NotFound desc = could not find container \"c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3\": container with ID starting with c440b39f210dac238d44ccf14c95f606d61e41cbf3c6e818d72b50a7966ab7a3 not found: ID does not exist" Nov 03 22:54:57.237032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.236986 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr"] Nov 03 22:54:57.283353 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.283325 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fb4f0a-7f6d59c45f-gpbjr"] Nov 03 22:54:57.340916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.340788 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229"] Nov 03 22:54:57.345311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.345291 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb4f0a-6fdc7bbbfd-m9229"] Nov 03 22:54:57.495448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.495416 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5ffd82-72be-42b9-be1b-b7ac56057509" path="/var/lib/kubelet/pods/2f5ffd82-72be-42b9-be1b-b7ac56057509/volumes" Nov 03 22:54:57.495906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:54:57.495892 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee77aaff-2035-49ba-9d77-5f13a2ad4a30" path="/var/lib/kubelet/pods/ee77aaff-2035-49ba-9d77-5f13a2ad4a30/volumes" Nov 03 22:55:03.984651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:03.984614 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd"] Nov 03 22:55:04.111159 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:04.111123 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9"] Nov 03 22:55:08.985034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:08.984991 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" podUID="1f8b506e-c569-401d-a1e3-b2bc7ec06db2" containerName="payload-processing" containerID="cri-o://665982b76d245edaa3eba3a55cbff62611f483ba54904b6f08448fced3b8823b" gracePeriod=25 Nov 03 22:55:09.112062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.112008 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" podUID="9180335f-26ce-4ff6-b046-a92e3f42303d" containerName="payload-pre-processing" containerID="cri-o://ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73" gracePeriod=25 Nov 03 22:55:09.198038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.197995 5272 generic.go:356] "Generic (PLEG): container finished" podID="1f8b506e-c569-401d-a1e3-b2bc7ec06db2" containerID="665982b76d245edaa3eba3a55cbff62611f483ba54904b6f08448fced3b8823b" exitCode=0 Nov 03 22:55:09.198223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.198060 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" event={"ID":"1f8b506e-c569-401d-a1e3-b2bc7ec06db2","Type":"ContainerDied","Data":"665982b76d245edaa3eba3a55cbff62611f483ba54904b6f08448fced3b8823b"} Nov 03 22:55:09.286663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.286635 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:55:09.405233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.405207 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-plugins-config-volume\") pod \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " Nov 03 22:55:09.405369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.405309 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-kube-api-access-tm8qm\" (UniqueName: \"kubernetes.io/projected/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-kube-api-access-tm8qm\") pod \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\" (UID: \"1f8b506e-c569-401d-a1e3-b2bc7ec06db2\") " Nov 03 22:55:09.405564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.405542 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-plugins-config-volume" pod "1f8b506e-c569-401d-a1e3-b2bc7ec06db2" (UID: "1f8b506e-c569-401d-a1e3-b2bc7ec06db2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:09.406117 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.406063 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:55:09.406259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.406115 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:09.407870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.407847 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-kube-api-access-tm8qm" pod "1f8b506e-c569-401d-a1e3-b2bc7ec06db2" (UID: "1f8b506e-c569-401d-a1e3-b2bc7ec06db2"). InnerVolumeSpecName "kube-api-access-tm8qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:09.508183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.508103 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9180335f-26ce-4ff6-b046-a92e3f42303d-kube-api-access-gt8bn\" (UniqueName: \"kubernetes.io/projected/9180335f-26ce-4ff6-b046-a92e3f42303d-kube-api-access-gt8bn\") pod \"9180335f-26ce-4ff6-b046-a92e3f42303d\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " Nov 03 22:55:09.508587 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.508563 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9180335f-26ce-4ff6-b046-a92e3f42303d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9180335f-26ce-4ff6-b046-a92e3f42303d-plugins-config-volume\") pod \"9180335f-26ce-4ff6-b046-a92e3f42303d\" (UID: \"9180335f-26ce-4ff6-b046-a92e3f42303d\") " Nov 03 22:55:09.510288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.510265 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm8qm\" (UniqueName: \"kubernetes.io/projected/1f8b506e-c569-401d-a1e3-b2bc7ec06db2-kube-api-access-tm8qm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:09.510604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.510578 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9180335f-26ce-4ff6-b046-a92e3f42303d-plugins-config-volume" pod "9180335f-26ce-4ff6-b046-a92e3f42303d" (UID: "9180335f-26ce-4ff6-b046-a92e3f42303d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:09.512256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.512228 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9180335f-26ce-4ff6-b046-a92e3f42303d-kube-api-access-gt8bn" pod "9180335f-26ce-4ff6-b046-a92e3f42303d" (UID: "9180335f-26ce-4ff6-b046-a92e3f42303d"). InnerVolumeSpecName "kube-api-access-gt8bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:09.612504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.612471 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8bn\" (UniqueName: \"kubernetes.io/projected/9180335f-26ce-4ff6-b046-a92e3f42303d-kube-api-access-gt8bn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:09.612504 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:09.612501 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9180335f-26ce-4ff6-b046-a92e3f42303d-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:10.208019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.207933 5272 generic.go:356] "Generic (PLEG): container finished" podID="9180335f-26ce-4ff6-b046-a92e3f42303d" containerID="ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73" exitCode=0 Nov 03 22:55:10.208019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.207994 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" Nov 03 22:55:10.208527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.208024 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" event={"ID":"9180335f-26ce-4ff6-b046-a92e3f42303d","Type":"ContainerDied","Data":"ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73"} Nov 03 22:55:10.208527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.208113 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9" event={"ID":"9180335f-26ce-4ff6-b046-a92e3f42303d","Type":"ContainerDied","Data":"38f9683b6925b02f1832c8fb124029d0c5cb4a5b4aa75bee96c9b476aa2a4a8f"} Nov 03 22:55:10.208527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.208134 5272 scope.go:122] "RemoveContainer" containerID="ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73" Nov 03 22:55:10.209712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.209693 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" Nov 03 22:55:10.209798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.209706 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd" event={"ID":"1f8b506e-c569-401d-a1e3-b2bc7ec06db2","Type":"ContainerDied","Data":"f659a71f2ed0a9366a80f066b7757cb908bf89a005022ca3c0f445a8c57db8d3"} Nov 03 22:55:10.220293 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.220261 5272 scope.go:122] "RemoveContainer" containerID="ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73" Nov 03 22:55:10.220541 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:10.220523 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73\": container with ID starting with ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73 not found: ID does not exist" containerID="ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73" Nov 03 22:55:10.220601 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.220547 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73"} err="failed to get container status \"ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73\": rpc error: code = NotFound desc = could not find container \"ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73\": container with ID starting with ef08bcf225c93c0039b3ddfa2a3e7f8d05e36651dd236a69e24388c27d0b6f73 not found: ID does not exist" Nov 03 22:55:10.220601 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.220565 5272 scope.go:122] "RemoveContainer" containerID="665982b76d245edaa3eba3a55cbff62611f483ba54904b6f08448fced3b8823b" Nov 03 22:55:10.363344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.363313 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9"] Nov 03 22:55:10.409549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.409517 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6a3169-67b8ff4bb7-k6wb9"] Nov 03 22:55:10.469699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.469584 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd"] Nov 03 22:55:10.474525 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.474499 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6a3169-866cf6cb6d-t7spd"] Nov 03 22:55:10.904674 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:10.904638 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:55:11.494623 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:11.494589 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8b506e-c569-401d-a1e3-b2bc7ec06db2" path="/var/lib/kubelet/pods/1f8b506e-c569-401d-a1e3-b2bc7ec06db2/volumes" Nov 03 22:55:11.495095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:11.495078 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9180335f-26ce-4ff6-b046-a92e3f42303d" path="/var/lib/kubelet/pods/9180335f-26ce-4ff6-b046-a92e3f42303d/volumes" Nov 03 22:55:11.640010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:11.639965 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc"] Nov 03 22:55:11.640270 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:11.640234 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" containerID="cri-o://98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d" gracePeriod=30 Nov 03 22:55:16.905648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:16.905624 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:55:17.029034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.028964 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-certs\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029003 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-credential-socket\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029028 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-data\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029060 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-token\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029174 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-kube-api-access-bvbwm\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-kube-api-access-bvbwm\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029212 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b12e991-5375-41b5-90ca-35b6f7bb2678-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e991-5375-41b5-90ca-35b6f7bb2678-istiod-ca-cert\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029226 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-certs" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.029299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029254 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-socket\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029299 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029299 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-podinfo\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029301 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-credential-socket" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.029613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029348 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-envoy\") pod \"8b12e991-5375-41b5-90ca-35b6f7bb2678\" (UID: \"8b12e991-5375-41b5-90ca-35b6f7bb2678\") " Nov 03 22:55:17.029613 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029534 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-socket" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.029771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029611 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b12e991-5375-41b5-90ca-35b6f7bb2678-istiod-ca-cert" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:17.029771 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.029711 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-data" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.030050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.030029 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e991-5375-41b5-90ca-35b6f7bb2678-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.030122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.030053 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.030122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.030067 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.030122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.030080 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.030122 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.030093 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.031682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.031655 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-token" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.031764 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.031678 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-kube-api-access-bvbwm" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "kube-api-access-bvbwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:17.031865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.031848 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-envoy" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:17.031962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.031949 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-podinfo" pod "8b12e991-5375-41b5-90ca-35b6f7bb2678" (UID: "8b12e991-5375-41b5-90ca-35b6f7bb2678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:17.131855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.131825 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.131855 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.131858 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.132012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.131871 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvbwm\" (UniqueName: \"kubernetes.io/projected/8b12e991-5375-41b5-90ca-35b6f7bb2678-kube-api-access-bvbwm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.132012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.131881 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e991-5375-41b5-90ca-35b6f7bb2678-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:17.271474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.271440 5272 generic.go:356] "Generic (PLEG): container finished" podID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerID="98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d" exitCode=0 Nov 03 22:55:17.271667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.271482 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" Nov 03 22:55:17.271667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.271508 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" event={"ID":"8b12e991-5375-41b5-90ca-35b6f7bb2678","Type":"ContainerDied","Data":"98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d"} Nov 03 22:55:17.271667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.271572 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc" event={"ID":"8b12e991-5375-41b5-90ca-35b6f7bb2678","Type":"ContainerDied","Data":"2601afa66d90b1997bfee1981bf0e8943cdffcbd567acc8b3824eef2e9515975"} Nov 03 22:55:17.271667 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.271601 5272 scope.go:122] "RemoveContainer" containerID="98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d" Nov 03 22:55:17.283896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.283879 5272 scope.go:122] "RemoveContainer" containerID="98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d" Nov 03 22:55:17.284143 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:17.284124 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d\": container with ID starting with 98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d not found: ID does not exist" containerID="98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d" Nov 03 22:55:17.284196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.284148 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d"} err="failed to get container status \"98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d\": rpc error: code = NotFound desc = could not find container \"98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d\": container with ID starting with 98a76ac9f6931889d24e4b2b4658e3b2c74accb9d4b5205f46f04be45be3fd4d not found: ID does not exist" Nov 03 22:55:17.427512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.427343 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc"] Nov 03 22:55:17.431157 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.431127 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-vhkwc"] Nov 03 22:55:17.498036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:17.498002 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" path="/var/lib/kubelet/pods/8b12e991-5375-41b5-90ca-35b6f7bb2678/volumes" Nov 03 22:55:24.560953 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:24.560856 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:55:25.407555 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.407514 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6"] Nov 03 22:55:25.408361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408333 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1f8b506e-c569-401d-a1e3-b2bc7ec06db2" containerName="payload-processing" Nov 03 22:55:25.408361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408355 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8b506e-c569-401d-a1e3-b2bc7ec06db2" containerName="payload-processing" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408370 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9180335f-26ce-4ff6-b046-a92e3f42303d" containerName="payload-pre-processing" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408378 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9180335f-26ce-4ff6-b046-a92e3f42303d" containerName="payload-pre-processing" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408405 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2f5ffd82-72be-42b9-be1b-b7ac56057509" containerName="payload-processing" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408411 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5ffd82-72be-42b9-be1b-b7ac56057509" containerName="payload-processing" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408441 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ca06d0f3-bdeb-4d9a-8d71-479d949be05e" containerName="revoke-keys" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408447 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca06d0f3-bdeb-4d9a-8d71-479d949be05e" containerName="revoke-keys" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408461 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408465 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408476 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ee77aaff-2035-49ba-9d77-5f13a2ad4a30" containerName="payload-pre-processing" Nov 03 22:55:25.408557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408481 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee77aaff-2035-49ba-9d77-5f13a2ad4a30" containerName="payload-pre-processing" Nov 03 22:55:25.408916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408590 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9180335f-26ce-4ff6-b046-a92e3f42303d" containerName="payload-pre-processing" Nov 03 22:55:25.408916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408606 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5ffd82-72be-42b9-be1b-b7ac56057509" containerName="payload-processing" Nov 03 22:55:25.408916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408623 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca06d0f3-bdeb-4d9a-8d71-479d949be05e" containerName="revoke-keys" Nov 03 22:55:25.408916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408638 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b12e991-5375-41b5-90ca-35b6f7bb2678" containerName="istio-proxy" Nov 03 22:55:25.408916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408647 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee77aaff-2035-49ba-9d77-5f13a2ad4a30" containerName="payload-pre-processing" Nov 03 22:55:25.408916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.408659 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8b506e-c569-401d-a1e3-b2bc7ec06db2" containerName="payload-processing" Nov 03 22:55:25.414015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.413999 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.465745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.465711 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29n8\" (UniqueName: \"kubernetes.io/projected/f1d553e1-48aa-4209-8724-656856b00bbf-kube-api-access-g29n8\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.466047 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.466003 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.466136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.466120 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.466762 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.466741 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6"] Nov 03 22:55:25.508392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.508354 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:55:25.508392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.508357 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:55:25.508392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.508383 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:55:25.508937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.508922 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:55:25.567582 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.567542 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.568087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.567634 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.568087 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.567752 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29n8\" (UniqueName: \"kubernetes.io/projected/f1d553e1-48aa-4209-8724-656856b00bbf-kube-api-access-g29n8\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.568297 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.568266 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.570260 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.570240 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.624677 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.624645 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29n8\" (UniqueName: \"kubernetes.io/projected/f1d553e1-48aa-4209-8724-656856b00bbf-kube-api-access-g29n8\") pod \"maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:25.728057 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:25.727974 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:26.106385 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:26.106353 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6"] Nov 03 22:55:26.112296 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:55:26.112257 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d553e1_48aa_4209_8724_656856b00bbf.slice/crio-fdf51f80154186bf64f25013a375ab9761564493de0afce9123adc13073f7717 WatchSource:0}: Error finding container fdf51f80154186bf64f25013a375ab9761564493de0afce9123adc13073f7717: Status 404 returned error can't find the container with id fdf51f80154186bf64f25013a375ab9761564493de0afce9123adc13073f7717 Nov 03 22:55:26.354997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:26.354969 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" event={"ID":"f1d553e1-48aa-4209-8724-656856b00bbf","Type":"ContainerStarted","Data":"fdf51f80154186bf64f25013a375ab9761564493de0afce9123adc13073f7717"} Nov 03 22:55:27.078168 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:27.078131 5272 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d553e1_48aa_4209_8724_656856b00bbf.slice/crio-conmon-c450ed4d0ae725e5b300981290b232cba048f9911677fb22a6f0756d5c093c64.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:55:27.078463 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:27.078260 5272 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d553e1_48aa_4209_8724_656856b00bbf.slice/crio-c450ed4d0ae725e5b300981290b232cba048f9911677fb22a6f0756d5c093c64.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:55:27.364301 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:27.364215 5272 generic.go:356] "Generic (PLEG): container finished" podID="f1d553e1-48aa-4209-8724-656856b00bbf" containerID="c450ed4d0ae725e5b300981290b232cba048f9911677fb22a6f0756d5c093c64" exitCode=0 Nov 03 22:55:27.364447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:27.364296 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" event={"ID":"f1d553e1-48aa-4209-8724-656856b00bbf","Type":"ContainerDied","Data":"c450ed4d0ae725e5b300981290b232cba048f9911677fb22a6f0756d5c093c64"} Nov 03 22:55:28.532133 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.532110 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:28.606969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.606937 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d553e1-48aa-4209-8724-656856b00bbf-kube-api-access-g29n8\" (UniqueName: \"kubernetes.io/projected/f1d553e1-48aa-4209-8724-656856b00bbf-kube-api-access-g29n8\") pod \"f1d553e1-48aa-4209-8724-656856b00bbf\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " Nov 03 22:55:28.607152 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.607133 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-openshift-ca\") pod \"f1d553e1-48aa-4209-8724-656856b00bbf\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " Nov 03 22:55:28.607230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.607214 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-xks-ca\") pod \"f1d553e1-48aa-4209-8724-656856b00bbf\" (UID: \"f1d553e1-48aa-4209-8724-656856b00bbf\") " Nov 03 22:55:28.607535 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.607504 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-openshift-ca" pod "f1d553e1-48aa-4209-8724-656856b00bbf" (UID: "f1d553e1-48aa-4209-8724-656856b00bbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:28.607845 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.607826 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:28.609259 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.609237 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-xks-ca" pod "f1d553e1-48aa-4209-8724-656856b00bbf" (UID: "f1d553e1-48aa-4209-8724-656856b00bbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:55:28.609493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.609479 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d553e1-48aa-4209-8724-656856b00bbf-kube-api-access-g29n8" pod "f1d553e1-48aa-4209-8724-656856b00bbf" (UID: "f1d553e1-48aa-4209-8724-656856b00bbf"). InnerVolumeSpecName "kube-api-access-g29n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:28.709544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.709465 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d553e1-48aa-4209-8724-656856b00bbf-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:28.709544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:28.709496 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29n8\" (UniqueName: \"kubernetes.io/projected/f1d553e1-48aa-4209-8724-656856b00bbf-kube-api-access-g29n8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:29.382244 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:29.382212 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" Nov 03 22:55:29.382457 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:29.382249 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6" event={"ID":"f1d553e1-48aa-4209-8724-656856b00bbf","Type":"ContainerDied","Data":"fdf51f80154186bf64f25013a375ab9761564493de0afce9123adc13073f7717"} Nov 03 22:55:29.382457 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:29.382328 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf51f80154186bf64f25013a375ab9761564493de0afce9123adc13073f7717" Nov 03 22:55:38.000986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:38.000954 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv"] Nov 03 22:55:38.104970 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:38.104939 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr"] Nov 03 22:55:42.727890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:42.727852 5272 scope.go:122] "RemoveContainer" containerID="da412c8f7c856588990f86684503f9b5406cda99febf8cc208904aa87bfa4e24" Nov 03 22:55:43.001519 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.001429 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" podUID="cc0de1f4-768a-4c03-b2fa-db92db8d8b69" containerName="payload-processing" containerID="cri-o://d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d" gracePeriod=25 Nov 03 22:55:43.106221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.106181 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" podUID="8c66265a-7fb0-4516-b53c-ecf85bb84806" containerName="payload-pre-processing" containerID="cri-o://adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2" gracePeriod=25 Nov 03 22:55:43.299399 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.299060 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:55:43.373759 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.373730 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-kube-api-access-x7l4s\" (UniqueName: \"kubernetes.io/projected/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-kube-api-access-x7l4s\") pod \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " Nov 03 22:55:43.373912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.373875 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-plugins-config-volume\") pod \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\" (UID: \"cc0de1f4-768a-4c03-b2fa-db92db8d8b69\") " Nov 03 22:55:43.374199 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.374177 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-plugins-config-volume" pod "cc0de1f4-768a-4c03-b2fa-db92db8d8b69" (UID: "cc0de1f4-768a-4c03-b2fa-db92db8d8b69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:43.374380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.374362 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:43.375713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.375694 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-kube-api-access-x7l4s" pod "cc0de1f4-768a-4c03-b2fa-db92db8d8b69" (UID: "cc0de1f4-768a-4c03-b2fa-db92db8d8b69"). InnerVolumeSpecName "kube-api-access-x7l4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:43.395833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.395813 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:55:43.475068 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.475038 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c66265a-7fb0-4516-b53c-ecf85bb84806-kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/8c66265a-7fb0-4516-b53c-ecf85bb84806-kube-api-access-hznjh\") pod \"8c66265a-7fb0-4516-b53c-ecf85bb84806\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " Nov 03 22:55:43.475213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.475093 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c66265a-7fb0-4516-b53c-ecf85bb84806-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c66265a-7fb0-4516-b53c-ecf85bb84806-plugins-config-volume\") pod \"8c66265a-7fb0-4516-b53c-ecf85bb84806\" (UID: \"8c66265a-7fb0-4516-b53c-ecf85bb84806\") " Nov 03 22:55:43.475462 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.475441 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c66265a-7fb0-4516-b53c-ecf85bb84806-plugins-config-volume" pod "8c66265a-7fb0-4516-b53c-ecf85bb84806" (UID: "8c66265a-7fb0-4516-b53c-ecf85bb84806"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:43.475552 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.475539 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7l4s\" (UniqueName: \"kubernetes.io/projected/cc0de1f4-768a-4c03-b2fa-db92db8d8b69-kube-api-access-x7l4s\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:43.475600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.475554 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c66265a-7fb0-4516-b53c-ecf85bb84806-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:43.477118 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.477094 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c66265a-7fb0-4516-b53c-ecf85bb84806-kube-api-access-hznjh" pod "8c66265a-7fb0-4516-b53c-ecf85bb84806" (UID: "8c66265a-7fb0-4516-b53c-ecf85bb84806"). InnerVolumeSpecName "kube-api-access-hznjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:43.576900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.576856 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/8c66265a-7fb0-4516-b53c-ecf85bb84806-kube-api-access-hznjh\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:55:43.624745 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.624717 5272 generic.go:356] "Generic (PLEG): container finished" podID="cc0de1f4-768a-4c03-b2fa-db92db8d8b69" containerID="d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d" exitCode=0 Nov 03 22:55:43.624923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.624755 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" Nov 03 22:55:43.624923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.624832 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" event={"ID":"cc0de1f4-768a-4c03-b2fa-db92db8d8b69","Type":"ContainerDied","Data":"d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d"} Nov 03 22:55:43.624923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.624882 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv" event={"ID":"cc0de1f4-768a-4c03-b2fa-db92db8d8b69","Type":"ContainerDied","Data":"3fb596510b58f39655df8c6ad7e8f885cf2ac6fd86e20f19615b123f6ef4e065"} Nov 03 22:55:43.624923 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.624908 5272 scope.go:122] "RemoveContainer" containerID="d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d" Nov 03 22:55:43.626250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.626231 5272 generic.go:356] "Generic (PLEG): container finished" podID="8c66265a-7fb0-4516-b53c-ecf85bb84806" containerID="adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2" exitCode=0 Nov 03 22:55:43.626369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.626256 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" event={"ID":"8c66265a-7fb0-4516-b53c-ecf85bb84806","Type":"ContainerDied","Data":"adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2"} Nov 03 22:55:43.626369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.626309 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" event={"ID":"8c66265a-7fb0-4516-b53c-ecf85bb84806","Type":"ContainerDied","Data":"e7e59fcb4c3a6a2b0c21513a559cef6bcb54709d653f8056bd7a02b73eff1335"} Nov 03 22:55:43.626369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.626268 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr" Nov 03 22:55:43.636092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.636072 5272 scope.go:122] "RemoveContainer" containerID="d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d" Nov 03 22:55:43.636347 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:43.636326 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d\": container with ID starting with d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d not found: ID does not exist" containerID="d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d" Nov 03 22:55:43.636428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.636354 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d"} err="failed to get container status \"d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d\": rpc error: code = NotFound desc = could not find container \"d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d\": container with ID starting with d8a2814b3f2ed52b6a2740b7f4e758ec0627a73c92e9778693fffde96cb4982d not found: ID does not exist" Nov 03 22:55:43.636428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.636371 5272 scope.go:122] "RemoveContainer" containerID="adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2" Nov 03 22:55:43.645710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.645690 5272 scope.go:122] "RemoveContainer" containerID="adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2" Nov 03 22:55:43.645967 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:43.645950 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2\": container with ID starting with adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2 not found: ID does not exist" containerID="adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2" Nov 03 22:55:43.646028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.645971 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2"} err="failed to get container status \"adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2\": rpc error: code = NotFound desc = could not find container \"adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2\": container with ID starting with adf6c8b275aeec820c696a93fc7461fc4bcd7fc6ee452b003179adae1fb0ded2 not found: ID does not exist" Nov 03 22:55:43.780596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.780573 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv"] Nov 03 22:55:43.784608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.784587 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c21c7-884dbcbfd-vcsmv"] Nov 03 22:55:43.886448 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.886393 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr"] Nov 03 22:55:43.892714 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:43.892695 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c21c7-dd7b5dd55-qzcqr"] Nov 03 22:55:45.494479 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:45.494443 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c66265a-7fb0-4516-b53c-ecf85bb84806" path="/var/lib/kubelet/pods/8c66265a-7fb0-4516-b53c-ecf85bb84806/volumes" Nov 03 22:55:45.495133 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:55:45.495044 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0de1f4-768a-4c03-b2fa-db92db8d8b69" path="/var/lib/kubelet/pods/cc0de1f4-768a-4c03-b2fa-db92db8d8b69/volumes" Nov 03 22:55:54.500372 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:55:54.500339 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:56:01.138742 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:01.138704 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:56:16.397189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:16.397146 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k"] Nov 03 22:56:16.432113 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:16.432073 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-922795-r6r4k"] Nov 03 22:56:17.536827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:17.536420 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2327346f-cdc8-45fa-b48e-680d5345640a" path="/var/lib/kubelet/pods/2327346f-cdc8-45fa-b48e-680d5345640a/volumes" Nov 03 22:56:17.766618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:17.766583 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c"] Nov 03 22:56:17.813092 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:17.813013 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb4f0a-dx92c"] Nov 03 22:56:19.495308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:19.495275 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597" path="/var/lib/kubelet/pods/ac7fb5aa-8ef1-4c36-8bb3-14b59adf4597/volumes" Nov 03 22:56:24.529152 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:56:24.529114 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:56:25.369440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:25.369408 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt"] Nov 03 22:56:25.369680 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:25.369657 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" containerID="cri-o://96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187" gracePeriod=30 Nov 03 22:56:26.725953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.725832 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6"] Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726715 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f1d553e1-48aa-4209-8724-656856b00bbf" containerName="revoke-keys" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726733 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d553e1-48aa-4209-8724-656856b00bbf" containerName="revoke-keys" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726764 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8c66265a-7fb0-4516-b53c-ecf85bb84806" containerName="payload-pre-processing" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726769 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c66265a-7fb0-4516-b53c-ecf85bb84806" containerName="payload-pre-processing" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726780 5272 cpu_manager.go:414] "containerMap: removing container" podUID="cc0de1f4-768a-4c03-b2fa-db92db8d8b69" containerName="payload-processing" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726785 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0de1f4-768a-4c03-b2fa-db92db8d8b69" containerName="payload-processing" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726964 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c66265a-7fb0-4516-b53c-ecf85bb84806" containerName="payload-pre-processing" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726976 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0de1f4-768a-4c03-b2fa-db92db8d8b69" containerName="payload-processing" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.726992 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d553e1-48aa-4209-8724-656856b00bbf" containerName="revoke-keys" Nov 03 22:56:26.732871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.732078 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.750938 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.750913 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.751058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.750953 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs4xw\" (UniqueName: \"kubernetes.io/projected/069c76bb-7918-40f1-aaef-f4a134527013-kube-api-access-rs4xw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.751235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.751218 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.784616 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.784588 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6"] Nov 03 22:56:26.826774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.826738 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:26.826774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.826750 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:56:26.826972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.826871 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:56:26.827318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.827305 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:56:26.852529 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.852498 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.852664 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.852549 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.852664 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.852568 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs4xw\" (UniqueName: \"kubernetes.io/projected/069c76bb-7918-40f1-aaef-f4a134527013-kube-api-access-rs4xw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.853233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.853209 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.854951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.854931 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:26.909947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:26.909925 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs4xw\" (UniqueName: \"kubernetes.io/projected/069c76bb-7918-40f1-aaef-f4a134527013-kube-api-access-rs4xw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:27.046829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.046788 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:27.390620 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.390529 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9"] Nov 03 22:56:27.396588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.396566 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.445556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.445533 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9b4383-openshift-default-dockercfg-kvhbw\"" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470362 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-istio-token\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470422 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-envoy\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470487 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27d9423-3e14-44de-a112-3edc286ab845-istiod-ca-cert\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470701 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-data\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470736 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-credential-socket\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470776 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-certs\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470817 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27d9423-3e14-44de-a112-3edc286ab845-istio-podinfo\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470888 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnllg\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-kube-api-access-tnllg\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.488119 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.470926 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-socket\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.537897 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:56:27.537472 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069c76bb_7918_40f1_aaef_f4a134527013.slice/crio-5cc28bf95a3751cbd0eb524d57bb04f09d8ff49724cfdbfcfe2ce3463f762547 WatchSource:0}: Error finding container 5cc28bf95a3751cbd0eb524d57bb04f09d8ff49724cfdbfcfe2ce3463f762547: Status 404 returned error can't find the container with id 5cc28bf95a3751cbd0eb524d57bb04f09d8ff49724cfdbfcfe2ce3463f762547 Nov 03 22:56:27.543275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.543243 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9"] Nov 03 22:56:27.550413 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.550105 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6"] Nov 03 22:56:27.571828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.571784 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-certs\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.571965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.571843 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27d9423-3e14-44de-a112-3edc286ab845-istio-podinfo\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.571965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.571927 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnllg\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-kube-api-access-tnllg\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.571975 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-socket\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572026 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-istio-token\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572093 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572056 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-envoy\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572128 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27d9423-3e14-44de-a112-3edc286ab845-istiod-ca-cert\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572168 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-certs\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572365 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-data\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572478 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572404 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-credential-socket\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.572727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572697 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-credential-socket\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.573510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.572780 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27d9423-3e14-44de-a112-3edc286ab845-istiod-ca-cert\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.573510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.573140 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-data\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.573510 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.573228 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-socket\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.575451 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.575429 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-envoy\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.576050 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.576024 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27d9423-3e14-44de-a112-3edc286ab845-istio-podinfo\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.630414 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.630388 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-istio-token\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.630512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.630473 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnllg\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-kube-api-access-tnllg\") pod \"e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:27.712151 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:27.712073 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:28.022716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:28.022645 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" event={"ID":"069c76bb-7918-40f1-aaef-f4a134527013","Type":"ContainerStarted","Data":"e73b417521c6cc5fc2cd782db954547983639834e4bb5338a46acf446260fd5f"} Nov 03 22:56:28.022716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:28.022699 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" event={"ID":"069c76bb-7918-40f1-aaef-f4a134527013","Type":"ContainerStarted","Data":"5cc28bf95a3751cbd0eb524d57bb04f09d8ff49724cfdbfcfe2ce3463f762547"} Nov 03 22:56:28.133458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:28.133409 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" podStartSLOduration=2.133395784 podStartE2EDuration="2.133395784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:28.130500954 +0000 UTC m=+1968.752809260" watchObservedRunningTime="2026-11-03 22:56:28.133395784 +0000 UTC m=+1968.755704091" Nov 03 22:56:28.159590 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:28.159554 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9"] Nov 03 22:56:28.165054 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:56:28.165029 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27d9423_3e14_44de_a112_3edc286ab845.slice/crio-b05853dc65daa1290b3adb5da707f011dc07e9baa5166dd4e39caa767c10f367 WatchSource:0}: Error finding container b05853dc65daa1290b3adb5da707f011dc07e9baa5166dd4e39caa767c10f367: Status 404 returned error can't find the container with id b05853dc65daa1290b3adb5da707f011dc07e9baa5166dd4e39caa767c10f367 Nov 03 22:56:28.167776 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:28.167743 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:28.167908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:28.167836 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:28.167908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:28.167885 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:56:29.032146 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:29.032107 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" event={"ID":"c27d9423-3e14-44de-a112-3edc286ab845","Type":"ContainerStarted","Data":"409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1"} Nov 03 22:56:29.032627 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:29.032166 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" event={"ID":"c27d9423-3e14-44de-a112-3edc286ab845","Type":"ContainerStarted","Data":"b05853dc65daa1290b3adb5da707f011dc07e9baa5166dd4e39caa767c10f367"} Nov 03 22:56:29.033975 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:29.033949 5272 generic.go:356] "Generic (PLEG): container finished" podID="069c76bb-7918-40f1-aaef-f4a134527013" containerID="e73b417521c6cc5fc2cd782db954547983639834e4bb5338a46acf446260fd5f" exitCode=0 Nov 03 22:56:29.034095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:29.034033 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" event={"ID":"069c76bb-7918-40f1-aaef-f4a134527013","Type":"ContainerDied","Data":"e73b417521c6cc5fc2cd782db954547983639834e4bb5338a46acf446260fd5f"} Nov 03 22:56:29.147752 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:29.147707 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podStartSLOduration=2.147694469 podStartE2EDuration="2.147694469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:29.144515622 +0000 UTC m=+1969.766823930" watchObservedRunningTime="2026-11-03 22:56:29.147694469 +0000 UTC m=+1969.770002775" Nov 03 22:56:29.714023 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:29.713987 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:29.739090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:29.739058 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:30.041579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.041002 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:30.042008 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.041960 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:56:30.214576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.214553 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:30.312258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.312179 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-openshift-ca\") pod \"069c76bb-7918-40f1-aaef-f4a134527013\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " Nov 03 22:56:30.312422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.312340 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069c76bb-7918-40f1-aaef-f4a134527013-kube-api-access-rs4xw\" (UniqueName: \"kubernetes.io/projected/069c76bb-7918-40f1-aaef-f4a134527013-kube-api-access-rs4xw\") pod \"069c76bb-7918-40f1-aaef-f4a134527013\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " Nov 03 22:56:30.312422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.312392 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-xks-ca\") pod \"069c76bb-7918-40f1-aaef-f4a134527013\" (UID: \"069c76bb-7918-40f1-aaef-f4a134527013\") " Nov 03 22:56:30.312541 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.312504 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-openshift-ca" pod "069c76bb-7918-40f1-aaef-f4a134527013" (UID: "069c76bb-7918-40f1-aaef-f4a134527013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:30.313014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.312989 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.314472 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.314445 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-xks-ca" pod "069c76bb-7918-40f1-aaef-f4a134527013" (UID: "069c76bb-7918-40f1-aaef-f4a134527013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:30.314600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.314584 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069c76bb-7918-40f1-aaef-f4a134527013-kube-api-access-rs4xw" pod "069c76bb-7918-40f1-aaef-f4a134527013" (UID: "069c76bb-7918-40f1-aaef-f4a134527013"). InnerVolumeSpecName "kube-api-access-rs4xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:30.414702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.414676 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs4xw\" (UniqueName: \"kubernetes.io/projected/069c76bb-7918-40f1-aaef-f4a134527013-kube-api-access-rs4xw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.414702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.414705 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069c76bb-7918-40f1-aaef-f4a134527013-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.625539 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.625511 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:56:30.718128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718099 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-data\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718279 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718151 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-token\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-token\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718279 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718218 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-podinfo\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718340 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-envoy\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718369 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istiod-ca-cert\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718428 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718390 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-data" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:30.718588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718470 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-certs\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718529 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-socket\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718609 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-credential-socket\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718706 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718646 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-kube-api-access-c9pdw\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-kube-api-access-c9pdw\") pod \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\" (UID: \"ade70f21-d0ee-4c05-aec9-602d5f1e22eb\") " Nov 03 22:56:30.718822 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718725 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-certs" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:30.718884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718797 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-socket" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:30.718884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718849 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istiod-ca-cert" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:30.718985 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.718958 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-credential-socket" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:30.719348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.719325 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.719348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.719346 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.719521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.719355 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.719521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.719364 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.719521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.719372 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.720682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.720657 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-podinfo" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:30.720789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.720765 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-envoy" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:30.720858 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.720830 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-token" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:30.720929 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.720910 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-kube-api-access-c9pdw" pod "ade70f21-d0ee-4c05-aec9-602d5f1e22eb" (UID: "ade70f21-d0ee-4c05-aec9-602d5f1e22eb"). InnerVolumeSpecName "kube-api-access-c9pdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:30.820311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.820275 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pdw\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-kube-api-access-c9pdw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.820311 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.820310 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.820493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.820320 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.820493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:30.820329 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade70f21-d0ee-4c05-aec9-602d5f1e22eb-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:31.049909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.049882 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" Nov 03 22:56:31.050310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.049911 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6" event={"ID":"069c76bb-7918-40f1-aaef-f4a134527013","Type":"ContainerDied","Data":"5cc28bf95a3751cbd0eb524d57bb04f09d8ff49724cfdbfcfe2ce3463f762547"} Nov 03 22:56:31.050310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.049961 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cc28bf95a3751cbd0eb524d57bb04f09d8ff49724cfdbfcfe2ce3463f762547" Nov 03 22:56:31.051715 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.051693 5272 generic.go:356] "Generic (PLEG): container finished" podID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerID="96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187" exitCode=0 Nov 03 22:56:31.051827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.051752 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" event={"ID":"ade70f21-d0ee-4c05-aec9-602d5f1e22eb","Type":"ContainerDied","Data":"96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187"} Nov 03 22:56:31.051827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.051779 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" Nov 03 22:56:31.051827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.051812 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt" event={"ID":"ade70f21-d0ee-4c05-aec9-602d5f1e22eb","Type":"ContainerDied","Data":"692c6204e61a27a205c24354ec493cfb491ecf72be916d323634fdd9c5b34a1e"} Nov 03 22:56:31.051965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.051844 5272 scope.go:122] "RemoveContainer" containerID="96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187" Nov 03 22:56:31.062716 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.062700 5272 scope.go:122] "RemoveContainer" containerID="c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1" Nov 03 22:56:31.072263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.072246 5272 scope.go:122] "RemoveContainer" containerID="96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187" Nov 03 22:56:31.072512 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:56:31.072494 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187\": container with ID starting with 96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187 not found: ID does not exist" containerID="96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187" Nov 03 22:56:31.072589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.072524 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187"} err="failed to get container status \"96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187\": rpc error: code = NotFound desc = could not find container \"96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187\": container with ID starting with 96752f033b4a0db81058e56836cf05dad3c77e8dea6009c43e3467ab5a775187 not found: ID does not exist" Nov 03 22:56:31.072589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.072547 5272 scope.go:122] "RemoveContainer" containerID="c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1" Nov 03 22:56:31.072784 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:56:31.072767 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1\": container with ID starting with c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1 not found: ID does not exist" containerID="c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1" Nov 03 22:56:31.072840 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.072790 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1"} err="failed to get container status \"c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1\": rpc error: code = NotFound desc = could not find container \"c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1\": container with ID starting with c3c84c990c796f82da4cc8400547eb8288b84c8232287f6b34973eb6429d40a1 not found: ID does not exist" Nov 03 22:56:31.208219 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.208178 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt"] Nov 03 22:56:31.217949 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.217925 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb4f0a-openshift-default-7b7c76784f-ldhbt"] Nov 03 22:56:31.495073 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:31.495003 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" path="/var/lib/kubelet/pods/ade70f21-d0ee-4c05-aec9-602d5f1e22eb/volumes" Nov 03 22:56:35.462964 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.462928 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98"] Nov 03 22:56:35.584872 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.579370 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-6a3169-g6b98"] Nov 03 22:56:35.777024 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.776954 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp"] Nov 03 22:56:35.777859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.777828 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" Nov 03 22:56:35.777859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.777847 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" Nov 03 22:56:35.778016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.777866 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" Nov 03 22:56:35.778016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.777873 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" Nov 03 22:56:35.778016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.777890 5272 cpu_manager.go:414] "containerMap: removing container" podUID="069c76bb-7918-40f1-aaef-f4a134527013" containerName="revoke-keys" Nov 03 22:56:35.778016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.777895 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069c76bb-7918-40f1-aaef-f4a134527013" containerName="revoke-keys" Nov 03 22:56:35.778164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.778071 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="069c76bb-7918-40f1-aaef-f4a134527013" containerName="revoke-keys" Nov 03 22:56:35.778164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.778101 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" Nov 03 22:56:35.778164 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.778115 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade70f21-d0ee-4c05-aec9-602d5f1e22eb" containerName="istio-proxy" Nov 03 22:56:35.781838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.781821 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:35.847217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.847182 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9b4383\"" Nov 03 22:56:35.853265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.853234 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp"] Nov 03 22:56:35.968520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.936190 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9b4383-dockercfg-h9qpx\"" Nov 03 22:56:35.979141 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.939208 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqm2\" (UniqueName: \"kubernetes.io/projected/ff009dd7-805d-4475-9eea-b93e4ccc4737-kube-api-access-rmqm2\") pod \"payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:35.979324 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.979177 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff009dd7-805d-4475-9eea-b93e4ccc4737-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:35.988842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.988773 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn"] Nov 03 22:56:35.994857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:35.994744 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.033306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.033280 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn"] Nov 03 22:56:36.080404 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.080375 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ab87b2-6e1a-4c03-ac68-1f8733368037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.080521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.080442 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmr7\" (UniqueName: \"kubernetes.io/projected/77ab87b2-6e1a-4c03-ac68-1f8733368037-kube-api-access-jjmr7\") pod \"payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.080560 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.080537 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqm2\" (UniqueName: \"kubernetes.io/projected/ff009dd7-805d-4475-9eea-b93e4ccc4737-kube-api-access-rmqm2\") pod \"payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:36.080598 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.080576 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff009dd7-805d-4475-9eea-b93e4ccc4737-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:36.081246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.081226 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff009dd7-805d-4475-9eea-b93e4ccc4737-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:36.137539 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.137520 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqm2\" (UniqueName: \"kubernetes.io/projected/ff009dd7-805d-4475-9eea-b93e4ccc4737-kube-api-access-rmqm2\") pod \"payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:36.181349 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.181326 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ab87b2-6e1a-4c03-ac68-1f8733368037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.181476 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.181388 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmr7\" (UniqueName: \"kubernetes.io/projected/77ab87b2-6e1a-4c03-ac68-1f8733368037-kube-api-access-jjmr7\") pod \"payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.181985 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.181967 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ab87b2-6e1a-4c03-ac68-1f8733368037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.238505 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.238484 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmr7\" (UniqueName: \"kubernetes.io/projected/77ab87b2-6e1a-4c03-ac68-1f8733368037-kube-api-access-jjmr7\") pod \"payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.311258 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.311202 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:36.427896 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.427823 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:36.697883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.697792 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn"] Nov 03 22:56:36.703342 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:56:36.703306 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ab87b2_6e1a_4c03_ac68_1f8733368037.slice/crio-a42ff71deb17054e39c72c139411332ee22059b6f235511006691ec57feca35f WatchSource:0}: Error finding container a42ff71deb17054e39c72c139411332ee22059b6f235511006691ec57feca35f: Status 404 returned error can't find the container with id a42ff71deb17054e39c72c139411332ee22059b6f235511006691ec57feca35f Nov 03 22:56:36.811196 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:36.811170 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp"] Nov 03 22:56:36.816145 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:56:36.816121 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff009dd7_805d_4475_9eea_b93e4ccc4737.slice/crio-0716f24488a69b81106346bfa3d42a6d1c12b8196edbc908426762b900a6e1f5 WatchSource:0}: Error finding container 0716f24488a69b81106346bfa3d42a6d1c12b8196edbc908426762b900a6e1f5: Status 404 returned error can't find the container with id 0716f24488a69b81106346bfa3d42a6d1c12b8196edbc908426762b900a6e1f5 Nov 03 22:56:37.101604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.101565 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" event={"ID":"77ab87b2-6e1a-4c03-ac68-1f8733368037","Type":"ContainerStarted","Data":"feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728"} Nov 03 22:56:37.101769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.101628 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" event={"ID":"77ab87b2-6e1a-4c03-ac68-1f8733368037","Type":"ContainerStarted","Data":"a42ff71deb17054e39c72c139411332ee22059b6f235511006691ec57feca35f"} Nov 03 22:56:37.101769 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.101662 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:37.103286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.103254 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" event={"ID":"ff009dd7-805d-4475-9eea-b93e4ccc4737","Type":"ContainerStarted","Data":"c6a0392046a3d12de9f5559a41d3a5ee74c0bcae1c256dc400c70ee1eaf7aede"} Nov 03 22:56:37.103391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.103305 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" event={"ID":"ff009dd7-805d-4475-9eea-b93e4ccc4737","Type":"ContainerStarted","Data":"0716f24488a69b81106346bfa3d42a6d1c12b8196edbc908426762b900a6e1f5"} Nov 03 22:56:37.103391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.103335 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:37.213934 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.213893 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" podStartSLOduration=2.213879962 podStartE2EDuration="2.213879962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:37.211025446 +0000 UTC m=+1977.833333753" watchObservedRunningTime="2026-11-03 22:56:37.213879962 +0000 UTC m=+1977.836188299" Nov 03 22:56:37.321902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.321850 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" podStartSLOduration=2.321836341 podStartE2EDuration="2.321836341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:37.31995836 +0000 UTC m=+1977.942266663" watchObservedRunningTime="2026-11-03 22:56:37.321836341 +0000 UTC m=+1977.944144663" Nov 03 22:56:37.506436 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:37.506347 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca06d0f3-bdeb-4d9a-8d71-479d949be05e" path="/var/lib/kubelet/pods/ca06d0f3-bdeb-4d9a-8d71-479d949be05e/volumes" Nov 03 22:56:38.798458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:38.798421 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45"] Nov 03 22:56:38.904115 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:38.904078 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs"] Nov 03 22:56:42.791645 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:42.791613 5272 scope.go:122] "RemoveContainer" containerID="6b465e159ca0ff7ff0715d1a000cee9b7cd49f5e33a547eee2061b1b313ea0a0" Nov 03 22:56:43.798830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:43.798762 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" podUID="c97d1225-8a29-40fb-8c43-3f810ad551c2" containerName="payload-processing" containerID="cri-o://ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e" gracePeriod=25 Nov 03 22:56:43.905308 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:43.905251 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" podUID="8e894261-faf0-4e8a-8d88-ffb419dae171" containerName="payload-pre-processing" containerID="cri-o://a240c58cf3d2a5e5474d290b8ee6feafa2c69cfe57f4a6d37aaf715867686460" gracePeriod=25 Nov 03 22:56:44.081184 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.081158 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:56:44.101556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.101525 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97d1225-8a29-40fb-8c43-3f810ad551c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97d1225-8a29-40fb-8c43-3f810ad551c2-plugins-config-volume\") pod \"c97d1225-8a29-40fb-8c43-3f810ad551c2\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " Nov 03 22:56:44.101711 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.101695 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97d1225-8a29-40fb-8c43-3f810ad551c2-kube-api-access-vllx4\" (UniqueName: \"kubernetes.io/projected/c97d1225-8a29-40fb-8c43-3f810ad551c2-kube-api-access-vllx4\") pod \"c97d1225-8a29-40fb-8c43-3f810ad551c2\" (UID: \"c97d1225-8a29-40fb-8c43-3f810ad551c2\") " Nov 03 22:56:44.101941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.101916 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97d1225-8a29-40fb-8c43-3f810ad551c2-plugins-config-volume" pod "c97d1225-8a29-40fb-8c43-3f810ad551c2" (UID: "c97d1225-8a29-40fb-8c43-3f810ad551c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:44.102288 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.102260 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97d1225-8a29-40fb-8c43-3f810ad551c2-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:44.104521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.104497 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97d1225-8a29-40fb-8c43-3f810ad551c2-kube-api-access-vllx4" pod "c97d1225-8a29-40fb-8c43-3f810ad551c2" (UID: "c97d1225-8a29-40fb-8c43-3f810ad551c2"). InnerVolumeSpecName "kube-api-access-vllx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:44.159492 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.159451 5272 generic.go:356] "Generic (PLEG): container finished" podID="c97d1225-8a29-40fb-8c43-3f810ad551c2" containerID="ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e" exitCode=0 Nov 03 22:56:44.159657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.159507 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" event={"ID":"c97d1225-8a29-40fb-8c43-3f810ad551c2","Type":"ContainerDied","Data":"ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e"} Nov 03 22:56:44.159657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.159571 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" event={"ID":"c97d1225-8a29-40fb-8c43-3f810ad551c2","Type":"ContainerDied","Data":"f8ad8e4aa9c2d903098368a5ee0fa2a6fc27972e4b16d23575a48430479f41f6"} Nov 03 22:56:44.159657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.159517 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45" Nov 03 22:56:44.159657 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.159599 5272 scope.go:122] "RemoveContainer" containerID="ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e" Nov 03 22:56:44.162466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.162409 5272 generic.go:356] "Generic (PLEG): container finished" podID="8e894261-faf0-4e8a-8d88-ffb419dae171" containerID="a240c58cf3d2a5e5474d290b8ee6feafa2c69cfe57f4a6d37aaf715867686460" exitCode=0 Nov 03 22:56:44.162551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.162492 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" event={"ID":"8e894261-faf0-4e8a-8d88-ffb419dae171","Type":"ContainerDied","Data":"a240c58cf3d2a5e5474d290b8ee6feafa2c69cfe57f4a6d37aaf715867686460"} Nov 03 22:56:44.174286 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.174266 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:56:44.175882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.175866 5272 scope.go:122] "RemoveContainer" containerID="ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e" Nov 03 22:56:44.176136 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:56:44.176116 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e\": container with ID starting with ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e not found: ID does not exist" containerID="ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e" Nov 03 22:56:44.176183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.176144 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e"} err="failed to get container status \"ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e\": rpc error: code = NotFound desc = could not find container \"ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e\": container with ID starting with ef3258c3ff38a0af1cb45555f65525f68bc6408c451b43b964053205cb7ce76e not found: ID does not exist" Nov 03 22:56:44.202877 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.202851 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e894261-faf0-4e8a-8d88-ffb419dae171-kube-api-access-7v4bf\" (UniqueName: \"kubernetes.io/projected/8e894261-faf0-4e8a-8d88-ffb419dae171-kube-api-access-7v4bf\") pod \"8e894261-faf0-4e8a-8d88-ffb419dae171\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " Nov 03 22:56:44.202979 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.202895 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e894261-faf0-4e8a-8d88-ffb419dae171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e894261-faf0-4e8a-8d88-ffb419dae171-plugins-config-volume\") pod \"8e894261-faf0-4e8a-8d88-ffb419dae171\" (UID: \"8e894261-faf0-4e8a-8d88-ffb419dae171\") " Nov 03 22:56:44.203272 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.203246 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e894261-faf0-4e8a-8d88-ffb419dae171-plugins-config-volume" pod "8e894261-faf0-4e8a-8d88-ffb419dae171" (UID: "8e894261-faf0-4e8a-8d88-ffb419dae171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:44.203432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.203411 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e894261-faf0-4e8a-8d88-ffb419dae171-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:44.203501 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.203436 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllx4\" (UniqueName: \"kubernetes.io/projected/c97d1225-8a29-40fb-8c43-3f810ad551c2-kube-api-access-vllx4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:44.205003 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.204982 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e894261-faf0-4e8a-8d88-ffb419dae171-kube-api-access-7v4bf" pod "8e894261-faf0-4e8a-8d88-ffb419dae171" (UID: "8e894261-faf0-4e8a-8d88-ffb419dae171"). InnerVolumeSpecName "kube-api-access-7v4bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:44.296969 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.296938 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45"] Nov 03 22:56:44.301237 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.301171 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65913e-75fdbc7fd8-vgd45"] Nov 03 22:56:44.304583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:44.304549 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4bf\" (UniqueName: \"kubernetes.io/projected/8e894261-faf0-4e8a-8d88-ffb419dae171-kube-api-access-7v4bf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:45.171175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:45.171091 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" Nov 03 22:56:45.171175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:45.171137 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs" event={"ID":"8e894261-faf0-4e8a-8d88-ffb419dae171","Type":"ContainerDied","Data":"e9c77f66fbe48ace863a963ab3ede91c4bf08fa803b4767700c3485c3c516e59"} Nov 03 22:56:45.171652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:45.171204 5272 scope.go:122] "RemoveContainer" containerID="a240c58cf3d2a5e5474d290b8ee6feafa2c69cfe57f4a6d37aaf715867686460" Nov 03 22:56:45.325007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:45.324969 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs"] Nov 03 22:56:45.371440 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:45.371409 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65913e-568db8dd5f-msdbs"] Nov 03 22:56:45.495086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:45.495012 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e894261-faf0-4e8a-8d88-ffb419dae171" path="/var/lib/kubelet/pods/8e894261-faf0-4e8a-8d88-ffb419dae171/volumes" Nov 03 22:56:45.495418 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:45.495405 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97d1225-8a29-40fb-8c43-3f810ad551c2" path="/var/lib/kubelet/pods/c97d1225-8a29-40fb-8c43-3f810ad551c2/volumes" Nov 03 22:56:46.478236 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.478204 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln"] Nov 03 22:56:46.478968 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.478943 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c97d1225-8a29-40fb-8c43-3f810ad551c2" containerName="payload-processing" Nov 03 22:56:46.478968 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.478966 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97d1225-8a29-40fb-8c43-3f810ad551c2" containerName="payload-processing" Nov 03 22:56:46.479134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.478980 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8e894261-faf0-4e8a-8d88-ffb419dae171" containerName="payload-pre-processing" Nov 03 22:56:46.479134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.478986 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e894261-faf0-4e8a-8d88-ffb419dae171" containerName="payload-pre-processing" Nov 03 22:56:46.479227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.479155 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97d1225-8a29-40fb-8c43-3f810ad551c2" containerName="payload-processing" Nov 03 22:56:46.479227 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.479165 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e894261-faf0-4e8a-8d88-ffb419dae171" containerName="payload-pre-processing" Nov 03 22:56:46.484250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.484234 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.538147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.538109 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln"] Nov 03 22:56:46.543095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.543069 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.543213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.543112 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.543293 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.543233 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsp2\" (UniqueName: \"kubernetes.io/projected/ebdc955a-fa53-4214-90d2-c14f34606a61-kube-api-access-hwsp2\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.557758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.557732 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:46.557758 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.557736 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:56:46.557930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.557736 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:56:46.579729 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.579705 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:56:46.644494 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.644464 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.644647 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.644509 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.644647 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.644564 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsp2\" (UniqueName: \"kubernetes.io/projected/ebdc955a-fa53-4214-90d2-c14f34606a61-kube-api-access-hwsp2\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.645228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.645204 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.646890 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.646864 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.701588 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.701557 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsp2\" (UniqueName: \"kubernetes.io/projected/ebdc955a-fa53-4214-90d2-c14f34606a61-kube-api-access-hwsp2\") pod \"maas-api-revoke-keys-e2e-api-a-2627c5-rxzln\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:46.797996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:46.797973 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:47.173297 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:47.173186 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln"] Nov 03 22:56:47.178936 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:56:47.178904 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdc955a_fa53_4214_90d2_c14f34606a61.slice/crio-970c161b4694101b756f7d099c507ea1e1b192fe92e3b7b36de143230b3e0f39 WatchSource:0}: Error finding container 970c161b4694101b756f7d099c507ea1e1b192fe92e3b7b36de143230b3e0f39: Status 404 returned error can't find the container with id 970c161b4694101b756f7d099c507ea1e1b192fe92e3b7b36de143230b3e0f39 Nov 03 22:56:48.112446 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:48.112410 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 22:56:48.112900 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:48.112477 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 22:56:48.195514 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:48.195482 5272 generic.go:356] "Generic (PLEG): container finished" podID="ebdc955a-fa53-4214-90d2-c14f34606a61" containerID="4ac19b0ca79cb8293415acd4c96b8581189f586e6ae075486f65bb9d24e1f57d" exitCode=0 Nov 03 22:56:48.195675 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:48.195535 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" event={"ID":"ebdc955a-fa53-4214-90d2-c14f34606a61","Type":"ContainerDied","Data":"4ac19b0ca79cb8293415acd4c96b8581189f586e6ae075486f65bb9d24e1f57d"} Nov 03 22:56:48.195675 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:48.195593 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" event={"ID":"ebdc955a-fa53-4214-90d2-c14f34606a61","Type":"ContainerStarted","Data":"970c161b4694101b756f7d099c507ea1e1b192fe92e3b7b36de143230b3e0f39"} Nov 03 22:56:49.354936 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.354911 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:49.380264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.380240 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-openshift-ca\") pod \"ebdc955a-fa53-4214-90d2-c14f34606a61\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " Nov 03 22:56:49.380392 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.380376 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdc955a-fa53-4214-90d2-c14f34606a61-kube-api-access-hwsp2\" (UniqueName: \"kubernetes.io/projected/ebdc955a-fa53-4214-90d2-c14f34606a61-kube-api-access-hwsp2\") pod \"ebdc955a-fa53-4214-90d2-c14f34606a61\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " Nov 03 22:56:49.380546 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.380529 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-xks-ca\") pod \"ebdc955a-fa53-4214-90d2-c14f34606a61\" (UID: \"ebdc955a-fa53-4214-90d2-c14f34606a61\") " Nov 03 22:56:49.380611 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.380592 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-openshift-ca" pod "ebdc955a-fa53-4214-90d2-c14f34606a61" (UID: "ebdc955a-fa53-4214-90d2-c14f34606a61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:49.381090 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.381071 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:49.382456 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.382428 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-xks-ca" pod "ebdc955a-fa53-4214-90d2-c14f34606a61" (UID: "ebdc955a-fa53-4214-90d2-c14f34606a61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:49.382550 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.382525 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdc955a-fa53-4214-90d2-c14f34606a61-kube-api-access-hwsp2" pod "ebdc955a-fa53-4214-90d2-c14f34606a61" (UID: "ebdc955a-fa53-4214-90d2-c14f34606a61"). InnerVolumeSpecName "kube-api-access-hwsp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:49.482946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.482873 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsp2\" (UniqueName: \"kubernetes.io/projected/ebdc955a-fa53-4214-90d2-c14f34606a61-kube-api-access-hwsp2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:49.482946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:49.482909 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdc955a-fa53-4214-90d2-c14f34606a61-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:50.211935 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:50.211860 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" Nov 03 22:56:50.212128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:50.211850 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln" event={"ID":"ebdc955a-fa53-4214-90d2-c14f34606a61","Type":"ContainerDied","Data":"970c161b4694101b756f7d099c507ea1e1b192fe92e3b7b36de143230b3e0f39"} Nov 03 22:56:50.212128 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:50.211983 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="970c161b4694101b756f7d099c507ea1e1b192fe92e3b7b36de143230b3e0f39" Nov 03 22:56:52.204158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:52.204122 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml"] Nov 03 22:56:52.204653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:52.204359 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" containerID="cri-o://ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2" gracePeriod=30 Nov 03 22:56:53.109751 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.109709 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8"] Nov 03 22:56:53.110856 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.110796 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ebdc955a-fa53-4214-90d2-c14f34606a61" containerName="revoke-keys" Nov 03 22:56:53.111036 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.111019 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdc955a-fa53-4214-90d2-c14f34606a61" containerName="revoke-keys" Nov 03 22:56:53.111399 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.111382 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdc955a-fa53-4214-90d2-c14f34606a61" containerName="revoke-keys" Nov 03 22:56:53.116506 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.116483 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.142827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.141653 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8"] Nov 03 22:56:53.214551 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.214525 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:56:53.220275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.215022 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:56:53.220275 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.215627 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:53.241040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.241010 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5pdz\" (UniqueName: \"kubernetes.io/projected/62213f58-aaf7-4526-be69-8209a46b7212-kube-api-access-d5pdz\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.241221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.241120 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.241221 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.241205 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.252441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.252414 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:56:53.343207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.343168 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.343470 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.343235 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.343470 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.343468 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5pdz\" (UniqueName: \"kubernetes.io/projected/62213f58-aaf7-4526-be69-8209a46b7212-kube-api-access-d5pdz\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.343999 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.343972 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.345605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.345584 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.401914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.401843 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5pdz\" (UniqueName: \"kubernetes.io/projected/62213f58-aaf7-4526-be69-8209a46b7212-kube-api-access-d5pdz\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.527865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.527829 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:53.552549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.552521 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:56:53.921323 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:53.921268 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8"] Nov 03 22:56:53.934057 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:56:53.934022 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62213f58_aaf7_4526_be69_8209a46b7212.slice/crio-368d0a4f02744eb49d94728561e46c8c13f292ce97cc1c1c6c8aa8014d83ba7a WatchSource:0}: Error finding container 368d0a4f02744eb49d94728561e46c8c13f292ce97cc1c1c6c8aa8014d83ba7a: Status 404 returned error can't find the container with id 368d0a4f02744eb49d94728561e46c8c13f292ce97cc1c1c6c8aa8014d83ba7a Nov 03 22:56:54.287404 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:54.284775 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" event={"ID":"62213f58-aaf7-4526-be69-8209a46b7212","Type":"ContainerStarted","Data":"368d0a4f02744eb49d94728561e46c8c13f292ce97cc1c1c6c8aa8014d83ba7a"} Nov 03 22:56:54.501257 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:56:54.501219 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:56:55.293610 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:55.293578 5272 generic.go:356] "Generic (PLEG): container finished" podID="62213f58-aaf7-4526-be69-8209a46b7212" containerID="c1f119503b699a326cf4945dade3d693706a82c3c5c0a578d2c8087328b86803" exitCode=0 Nov 03 22:56:55.294016 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:55.293656 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" event={"ID":"62213f58-aaf7-4526-be69-8209a46b7212","Type":"ContainerDied","Data":"c1f119503b699a326cf4945dade3d693706a82c3c5c0a578d2c8087328b86803"} Nov 03 22:56:56.488596 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.488570 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:56.497828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.497794 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-openshift-ca\") pod \"62213f58-aaf7-4526-be69-8209a46b7212\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " Nov 03 22:56:56.497947 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.497935 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62213f58-aaf7-4526-be69-8209a46b7212-kube-api-access-d5pdz\" (UniqueName: \"kubernetes.io/projected/62213f58-aaf7-4526-be69-8209a46b7212-kube-api-access-d5pdz\") pod \"62213f58-aaf7-4526-be69-8209a46b7212\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " Nov 03 22:56:56.498006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.497993 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-xks-ca\") pod \"62213f58-aaf7-4526-be69-8209a46b7212\" (UID: \"62213f58-aaf7-4526-be69-8209a46b7212\") " Nov 03 22:56:56.498070 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.498048 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-openshift-ca" pod "62213f58-aaf7-4526-be69-8209a46b7212" (UID: "62213f58-aaf7-4526-be69-8209a46b7212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:56.498493 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.498474 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:56.499986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.499959 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-xks-ca" pod "62213f58-aaf7-4526-be69-8209a46b7212" (UID: "62213f58-aaf7-4526-be69-8209a46b7212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:56.500076 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.500004 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62213f58-aaf7-4526-be69-8209a46b7212-kube-api-access-d5pdz" pod "62213f58-aaf7-4526-be69-8209a46b7212" (UID: "62213f58-aaf7-4526-be69-8209a46b7212"). InnerVolumeSpecName "kube-api-access-d5pdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:56.599688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.599587 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62213f58-aaf7-4526-be69-8209a46b7212-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:56.599688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:56.599620 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5pdz\" (UniqueName: \"kubernetes.io/projected/62213f58-aaf7-4526-be69-8209a46b7212-kube-api-access-d5pdz\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.309365 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.309338 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" Nov 03 22:56:57.309583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.309344 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8" event={"ID":"62213f58-aaf7-4526-be69-8209a46b7212","Type":"ContainerDied","Data":"368d0a4f02744eb49d94728561e46c8c13f292ce97cc1c1c6c8aa8014d83ba7a"} Nov 03 22:56:57.309583 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.309465 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368d0a4f02744eb49d94728561e46c8c13f292ce97cc1c1c6c8aa8014d83ba7a" Nov 03 22:56:57.520792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.520759 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:56:57.614482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614452 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d448175a-3e0e-4b5d-b786-010d6a58a1af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d448175a-3e0e-4b5d-b786-010d6a58a1af-istiod-ca-cert\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.614624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614494 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-data\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.614624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614513 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-envoy\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.614624 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614556 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-kube-api-access-bdt7v\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-kube-api-access-bdt7v\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.614778 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614633 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-socket\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.614865 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614773 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-certs\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.614921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614898 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-credential-socket\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.614974 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614773 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-data" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:57.615012 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.614974 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d448175a-3e0e-4b5d-b786-010d6a58a1af-istiod-ca-cert" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:57.615071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.615023 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-certs" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:57.615071 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.615038 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-token\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-token\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.615178 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.615060 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-socket" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:57.615178 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.615121 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-podinfo\") pod \"d448175a-3e0e-4b5d-b786-010d6a58a1af\" (UID: \"d448175a-3e0e-4b5d-b786-010d6a58a1af\") " Nov 03 22:56:57.615290 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.615233 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-credential-socket" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:57.615994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.615964 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d448175a-3e0e-4b5d-b786-010d6a58a1af-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.616127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.615998 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.616127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.616012 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.616127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.616026 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.616127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.616039 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.617135 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.617108 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-envoy" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:57.617228 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.617182 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-podinfo" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:57.617357 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.617338 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-token" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:57.618038 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.618021 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-kube-api-access-bdt7v" pod "d448175a-3e0e-4b5d-b786-010d6a58a1af" (UID: "d448175a-3e0e-4b5d-b786-010d6a58a1af"). InnerVolumeSpecName "kube-api-access-bdt7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:57.718342 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.718312 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.718595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.718572 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.718724 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.718709 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d448175a-3e0e-4b5d-b786-010d6a58a1af-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:57.718857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:57.718840 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdt7v\" (UniqueName: \"kubernetes.io/projected/d448175a-3e0e-4b5d-b786-010d6a58a1af-kube-api-access-bdt7v\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:56:58.318137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.318102 5272 generic.go:356] "Generic (PLEG): container finished" podID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerID="ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2" exitCode=0 Nov 03 22:56:58.318350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.318161 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" Nov 03 22:56:58.318350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.318183 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" event={"ID":"d448175a-3e0e-4b5d-b786-010d6a58a1af","Type":"ContainerDied","Data":"ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2"} Nov 03 22:56:58.318350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.318238 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml" event={"ID":"d448175a-3e0e-4b5d-b786-010d6a58a1af","Type":"ContainerDied","Data":"b4f7288f9ca2f98e5fae24c2b72fa3229856d047c4d3ca196585f11e9e1ed92e"} Nov 03 22:56:58.318350 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.318262 5272 scope.go:122] "RemoveContainer" containerID="ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2" Nov 03 22:56:58.330184 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.330013 5272 scope.go:122] "RemoveContainer" containerID="f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb" Nov 03 22:56:58.339690 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.339673 5272 scope.go:122] "RemoveContainer" containerID="ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2" Nov 03 22:56:58.339974 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:56:58.339948 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2\": container with ID starting with ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2 not found: ID does not exist" containerID="ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2" Nov 03 22:56:58.340077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.339979 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2"} err="failed to get container status \"ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2\": rpc error: code = NotFound desc = could not find container \"ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2\": container with ID starting with ea17b45a6b11af3ba8381fcb041023424fb92c4a91f30f217311249277e164c2 not found: ID does not exist" Nov 03 22:56:58.340077 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.339998 5272 scope.go:122] "RemoveContainer" containerID="f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb" Nov 03 22:56:58.340248 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:56:58.340232 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb\": container with ID starting with f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb not found: ID does not exist" containerID="f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb" Nov 03 22:56:58.340296 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.340253 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb"} err="failed to get container status \"f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb\": rpc error: code = NotFound desc = could not find container \"f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb\": container with ID starting with f8af4005ac992683d037c700cada6e5ff87935edd2b6d02f20703560ed07d6fb not found: ID does not exist" Nov 03 22:56:58.475432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.475399 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml"] Nov 03 22:56:58.523842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.523763 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6a3169-openshift-default-58d9474d87-gztml"] Nov 03 22:56:58.636605 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.636520 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f"] Nov 03 22:56:58.777109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:58.777070 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t"] Nov 03 22:56:59.495026 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:56:59.494994 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" path="/var/lib/kubelet/pods/d448175a-3e0e-4b5d-b786-010d6a58a1af/volumes" Nov 03 22:57:00.833210 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.833175 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99"] Nov 03 22:57:00.833946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.833928 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" Nov 03 22:57:00.833946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.833947 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" Nov 03 22:57:00.834084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.833962 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" Nov 03 22:57:00.834084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.833967 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" Nov 03 22:57:00.834084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.833983 5272 cpu_manager.go:414] "containerMap: removing container" podUID="62213f58-aaf7-4526-be69-8209a46b7212" containerName="revoke-keys" Nov 03 22:57:00.834084 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.833989 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62213f58-aaf7-4526-be69-8209a46b7212" containerName="revoke-keys" Nov 03 22:57:00.834233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.834197 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" Nov 03 22:57:00.834233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.834209 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="62213f58-aaf7-4526-be69-8209a46b7212" containerName="revoke-keys" Nov 03 22:57:00.839389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.839374 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:00.888767 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.888734 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:57:00.892142 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.892115 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99"] Nov 03 22:57:00.915621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.915585 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:00.915798 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.915783 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:00.915931 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.915904 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6525w\" (UniqueName: \"kubernetes.io/projected/69f67821-8bb2-49f4-9b23-f42bb1706448-kube-api-access-6525w\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:00.934185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.934155 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:57:00.934343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.934231 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:57:00.934868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:00.934854 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:57:01.017916 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.017885 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:01.018066 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.017943 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6525w\" (UniqueName: \"kubernetes.io/projected/69f67821-8bb2-49f4-9b23-f42bb1706448-kube-api-access-6525w\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:01.018129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.018081 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:01.018723 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.018696 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:01.020261 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.020240 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:01.075269 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.075233 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6525w\" (UniqueName: \"kubernetes.io/projected/69f67821-8bb2-49f4-9b23-f42bb1706448-kube-api-access-6525w\") pod \"maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:01.153116 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.153034 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:01.568527 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:01.568495 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99"] Nov 03 22:57:01.574265 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:57:01.574230 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f67821_8bb2_49f4_9b23_f42bb1706448.slice/crio-b8b7b09cefa9dd6c87d24c016cde139c83553def698f11c5787a9519bb317c40 WatchSource:0}: Error finding container b8b7b09cefa9dd6c87d24c016cde139c83553def698f11c5787a9519bb317c40: Status 404 returned error can't find the container with id b8b7b09cefa9dd6c87d24c016cde139c83553def698f11c5787a9519bb317c40 Nov 03 22:57:02.403189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:02.403148 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" event={"ID":"69f67821-8bb2-49f4-9b23-f42bb1706448","Type":"ContainerStarted","Data":"d7aa15997e06f3d47a24717d4d7de21f5286210aff2160dac4cb8aba8761a227"} Nov 03 22:57:02.403592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:02.403217 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" event={"ID":"69f67821-8bb2-49f4-9b23-f42bb1706448","Type":"ContainerStarted","Data":"b8b7b09cefa9dd6c87d24c016cde139c83553def698f11c5787a9519bb317c40"} Nov 03 22:57:02.511726 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:02.511678 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" podStartSLOduration=2.511665172 podStartE2EDuration="2.511665172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:02.510600234 +0000 UTC m=+2003.132908542" watchObservedRunningTime="2026-11-03 22:57:02.511665172 +0000 UTC m=+2003.133973479" Nov 03 22:57:03.411642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.411606 5272 generic.go:356] "Generic (PLEG): container finished" podID="69f67821-8bb2-49f4-9b23-f42bb1706448" containerID="d7aa15997e06f3d47a24717d4d7de21f5286210aff2160dac4cb8aba8761a227" exitCode=0 Nov 03 22:57:03.412124 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.411652 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" event={"ID":"69f67821-8bb2-49f4-9b23-f42bb1706448","Type":"ContainerDied","Data":"d7aa15997e06f3d47a24717d4d7de21f5286210aff2160dac4cb8aba8761a227"} Nov 03 22:57:03.636907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.636863 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" podUID="176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" containerName="payload-processing" containerID="cri-o://884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686" gracePeriod=25 Nov 03 22:57:03.777606 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.777542 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" podUID="d0152dff-65b4-433c-a22a-9e049aa15d41" containerName="payload-pre-processing" containerID="cri-o://6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196" gracePeriod=25 Nov 03 22:57:03.910904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.910876 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:57:03.961925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.958600 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-plugins-config-volume\") pod \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " Nov 03 22:57:03.961925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.958752 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-kube-api-access-nwmz4\" (UniqueName: \"kubernetes.io/projected/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-kube-api-access-nwmz4\") pod \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\" (UID: \"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63\") " Nov 03 22:57:03.961925 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.961474 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-plugins-config-volume" pod "176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" (UID: "176f3cd2-99cf-4a3a-a2f7-db6c0da7df63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:03.966383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:03.966355 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-kube-api-access-nwmz4" pod "176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" (UID: "176f3cd2-99cf-4a3a-a2f7-db6c0da7df63"). InnerVolumeSpecName "kube-api-access-nwmz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:04.026000 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.025968 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:57:04.060859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.060818 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:04.060859 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.060854 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwmz4\" (UniqueName: \"kubernetes.io/projected/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63-kube-api-access-nwmz4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:04.162247 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.162211 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0152dff-65b4-433c-a22a-9e049aa15d41-kube-api-access-jwcfs\" (UniqueName: \"kubernetes.io/projected/d0152dff-65b4-433c-a22a-9e049aa15d41-kube-api-access-jwcfs\") pod \"d0152dff-65b4-433c-a22a-9e049aa15d41\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " Nov 03 22:57:04.162420 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.162314 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0152dff-65b4-433c-a22a-9e049aa15d41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0152dff-65b4-433c-a22a-9e049aa15d41-plugins-config-volume\") pod \"d0152dff-65b4-433c-a22a-9e049aa15d41\" (UID: \"d0152dff-65b4-433c-a22a-9e049aa15d41\") " Nov 03 22:57:04.162692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.162668 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0152dff-65b4-433c-a22a-9e049aa15d41-plugins-config-volume" pod "d0152dff-65b4-433c-a22a-9e049aa15d41" (UID: "d0152dff-65b4-433c-a22a-9e049aa15d41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:04.162921 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.162904 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0152dff-65b4-433c-a22a-9e049aa15d41-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:04.164447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.164420 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0152dff-65b4-433c-a22a-9e049aa15d41-kube-api-access-jwcfs" pod "d0152dff-65b4-433c-a22a-9e049aa15d41" (UID: "d0152dff-65b4-433c-a22a-9e049aa15d41"). InnerVolumeSpecName "kube-api-access-jwcfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:04.264549 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.264510 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwcfs\" (UniqueName: \"kubernetes.io/projected/d0152dff-65b4-433c-a22a-9e049aa15d41-kube-api-access-jwcfs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:04.421007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.420972 5272 generic.go:356] "Generic (PLEG): container finished" podID="176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" containerID="884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686" exitCode=0 Nov 03 22:57:04.421460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.421015 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" Nov 03 22:57:04.421460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.421057 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" event={"ID":"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63","Type":"ContainerDied","Data":"884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686"} Nov 03 22:57:04.421460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.421103 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f" event={"ID":"176f3cd2-99cf-4a3a-a2f7-db6c0da7df63","Type":"ContainerDied","Data":"26272c01d4f1676a378e34783b722311720ecc417cc3235679cdef1ac9284b9a"} Nov 03 22:57:04.421460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.421128 5272 scope.go:122] "RemoveContainer" containerID="884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686" Nov 03 22:57:04.422585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.422563 5272 generic.go:356] "Generic (PLEG): container finished" podID="d0152dff-65b4-433c-a22a-9e049aa15d41" containerID="6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196" exitCode=0 Nov 03 22:57:04.422683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.422607 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" Nov 03 22:57:04.422683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.422647 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" event={"ID":"d0152dff-65b4-433c-a22a-9e049aa15d41","Type":"ContainerDied","Data":"6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196"} Nov 03 22:57:04.422789 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.422695 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t" event={"ID":"d0152dff-65b4-433c-a22a-9e049aa15d41","Type":"ContainerDied","Data":"a8204937577d838c03fde10aebd72592ab5fd804762483a55a206eb6f53568ae"} Nov 03 22:57:04.433147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.433130 5272 scope.go:122] "RemoveContainer" containerID="884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686" Nov 03 22:57:04.433413 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:04.433394 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686\": container with ID starting with 884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686 not found: ID does not exist" containerID="884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686" Nov 03 22:57:04.433495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.433419 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686"} err="failed to get container status \"884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686\": rpc error: code = NotFound desc = could not find container \"884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686\": container with ID starting with 884fa453bccf2c7f8572fe77b095c350b68cc8d920c284785d4a026a72e6e686 not found: ID does not exist" Nov 03 22:57:04.433495 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.433433 5272 scope.go:122] "RemoveContainer" containerID="6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196" Nov 03 22:57:04.444926 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.444902 5272 scope.go:122] "RemoveContainer" containerID="6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196" Nov 03 22:57:04.445251 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:04.445220 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196\": container with ID starting with 6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196 not found: ID does not exist" containerID="6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196" Nov 03 22:57:04.445336 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.445258 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196"} err="failed to get container status \"6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196\": rpc error: code = NotFound desc = could not find container \"6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196\": container with ID starting with 6059153afed161fbc460b24f75ad9dfeeb125b75fa052fb019ff611a96d7a196 not found: ID does not exist" Nov 03 22:57:04.570461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.570435 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:04.576404 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.576379 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t"] Nov 03 22:57:04.627040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.627007 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2627c5-86d7c64d59-2l69t"] Nov 03 22:57:04.669499 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.669474 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-openshift-ca\") pod \"69f67821-8bb2-49f4-9b23-f42bb1706448\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " Nov 03 22:57:04.669625 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.669528 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-xks-ca\") pod \"69f67821-8bb2-49f4-9b23-f42bb1706448\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " Nov 03 22:57:04.669625 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.669556 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f67821-8bb2-49f4-9b23-f42bb1706448-kube-api-access-6525w\" (UniqueName: \"kubernetes.io/projected/69f67821-8bb2-49f4-9b23-f42bb1706448-kube-api-access-6525w\") pod \"69f67821-8bb2-49f4-9b23-f42bb1706448\" (UID: \"69f67821-8bb2-49f4-9b23-f42bb1706448\") " Nov 03 22:57:04.669906 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.669880 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-openshift-ca" pod "69f67821-8bb2-49f4-9b23-f42bb1706448" (UID: "69f67821-8bb2-49f4-9b23-f42bb1706448"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:04.670148 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.670127 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:04.671772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.671701 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-xks-ca" pod "69f67821-8bb2-49f4-9b23-f42bb1706448" (UID: "69f67821-8bb2-49f4-9b23-f42bb1706448"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:04.671772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.671746 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f67821-8bb2-49f4-9b23-f42bb1706448-kube-api-access-6525w" pod "69f67821-8bb2-49f4-9b23-f42bb1706448" (UID: "69f67821-8bb2-49f4-9b23-f42bb1706448"). InnerVolumeSpecName "kube-api-access-6525w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:04.684412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.684390 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f"] Nov 03 22:57:04.690490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.690466 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2627c5-56b96dd67b-vqv5f"] Nov 03 22:57:04.771021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.770995 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f67821-8bb2-49f4-9b23-f42bb1706448-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:04.771021 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.771021 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6525w\" (UniqueName: \"kubernetes.io/projected/69f67821-8bb2-49f4-9b23-f42bb1706448-kube-api-access-6525w\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:04.883441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.883410 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j"] Nov 03 22:57:04.997813 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:04.991575 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb"] Nov 03 22:57:05.435832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:05.435776 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" Nov 03 22:57:05.436226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:05.435775 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99" event={"ID":"69f67821-8bb2-49f4-9b23-f42bb1706448","Type":"ContainerDied","Data":"b8b7b09cefa9dd6c87d24c016cde139c83553def698f11c5787a9519bb317c40"} Nov 03 22:57:05.436226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:05.435917 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8b7b09cefa9dd6c87d24c016cde139c83553def698f11c5787a9519bb317c40" Nov 03 22:57:05.540513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:05.540477 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" path="/var/lib/kubelet/pods/176f3cd2-99cf-4a3a-a2f7-db6c0da7df63/volumes" Nov 03 22:57:05.541034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:05.541016 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0152dff-65b4-433c-a22a-9e049aa15d41" path="/var/lib/kubelet/pods/d0152dff-65b4-433c-a22a-9e049aa15d41/volumes" Nov 03 22:57:09.884337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:09.884294 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" podUID="6bb5cb89-168d-4eac-863d-a769432088fe" containerName="payload-processing" containerID="cri-o://f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94" gracePeriod=25 Nov 03 22:57:09.999303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:09.999262 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" podUID="2237d33f-a103-4ad7-908a-abaa7731658c" containerName="payload-pre-processing" containerID="cri-o://65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0" gracePeriod=25 Nov 03 22:57:10.165648 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.165617 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:57:10.272862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.272836 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:57:10.289910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.289880 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb5cb89-168d-4eac-863d-a769432088fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb5cb89-168d-4eac-863d-a769432088fe-plugins-config-volume\") pod \"6bb5cb89-168d-4eac-863d-a769432088fe\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " Nov 03 22:57:10.290058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.289965 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb5cb89-168d-4eac-863d-a769432088fe-kube-api-access-qzlbm\" (UniqueName: \"kubernetes.io/projected/6bb5cb89-168d-4eac-863d-a769432088fe-kube-api-access-qzlbm\") pod \"6bb5cb89-168d-4eac-863d-a769432088fe\" (UID: \"6bb5cb89-168d-4eac-863d-a769432088fe\") " Nov 03 22:57:10.290283 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.290257 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb5cb89-168d-4eac-863d-a769432088fe-plugins-config-volume" pod "6bb5cb89-168d-4eac-863d-a769432088fe" (UID: "6bb5cb89-168d-4eac-863d-a769432088fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:10.290585 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.290564 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb5cb89-168d-4eac-863d-a769432088fe-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:10.292537 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.292517 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb5cb89-168d-4eac-863d-a769432088fe-kube-api-access-qzlbm" pod "6bb5cb89-168d-4eac-863d-a769432088fe" (UID: "6bb5cb89-168d-4eac-863d-a769432088fe"). InnerVolumeSpecName "kube-api-access-qzlbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:10.391621 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.391542 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2237d33f-a103-4ad7-908a-abaa7731658c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2237d33f-a103-4ad7-908a-abaa7731658c-plugins-config-volume\") pod \"2237d33f-a103-4ad7-908a-abaa7731658c\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " Nov 03 22:57:10.391787 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.391625 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2237d33f-a103-4ad7-908a-abaa7731658c-kube-api-access-5bpw4\" (UniqueName: \"kubernetes.io/projected/2237d33f-a103-4ad7-908a-abaa7731658c-kube-api-access-5bpw4\") pod \"2237d33f-a103-4ad7-908a-abaa7731658c\" (UID: \"2237d33f-a103-4ad7-908a-abaa7731658c\") " Nov 03 22:57:10.391939 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.391915 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2237d33f-a103-4ad7-908a-abaa7731658c-plugins-config-volume" pod "2237d33f-a103-4ad7-908a-abaa7731658c" (UID: "2237d33f-a103-4ad7-908a-abaa7731658c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:10.392048 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.392033 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzlbm\" (UniqueName: \"kubernetes.io/projected/6bb5cb89-168d-4eac-863d-a769432088fe-kube-api-access-qzlbm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:10.392114 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.392050 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2237d33f-a103-4ad7-908a-abaa7731658c-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:10.393663 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.393633 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2237d33f-a103-4ad7-908a-abaa7731658c-kube-api-access-5bpw4" pod "2237d33f-a103-4ad7-908a-abaa7731658c" (UID: "2237d33f-a103-4ad7-908a-abaa7731658c"). InnerVolumeSpecName "kube-api-access-5bpw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:10.487548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.487516 5272 generic.go:356] "Generic (PLEG): container finished" podID="6bb5cb89-168d-4eac-863d-a769432088fe" containerID="f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94" exitCode=0 Nov 03 22:57:10.487740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.487551 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" Nov 03 22:57:10.487740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.487594 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" event={"ID":"6bb5cb89-168d-4eac-863d-a769432088fe","Type":"ContainerDied","Data":"f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94"} Nov 03 22:57:10.487740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.487644 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j" event={"ID":"6bb5cb89-168d-4eac-863d-a769432088fe","Type":"ContainerDied","Data":"c6ef96cf618f4b98b66ded53fa76dd931a05aea05abde2be7018196dda9da3d8"} Nov 03 22:57:10.487740 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.487673 5272 scope.go:122] "RemoveContainer" containerID="f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94" Nov 03 22:57:10.489058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.489039 5272 generic.go:356] "Generic (PLEG): container finished" podID="2237d33f-a103-4ad7-908a-abaa7731658c" containerID="65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0" exitCode=0 Nov 03 22:57:10.489177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.489074 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" Nov 03 22:57:10.489177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.489121 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" event={"ID":"2237d33f-a103-4ad7-908a-abaa7731658c","Type":"ContainerDied","Data":"65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0"} Nov 03 22:57:10.489177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.489173 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb" event={"ID":"2237d33f-a103-4ad7-908a-abaa7731658c","Type":"ContainerDied","Data":"13b5958629595b9834658a936e4fedc8cafe3217bf3aff3321342d566d3637f8"} Nov 03 22:57:10.492727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.492702 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpw4\" (UniqueName: \"kubernetes.io/projected/2237d33f-a103-4ad7-908a-abaa7731658c-kube-api-access-5bpw4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:10.498361 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.498298 5272 scope.go:122] "RemoveContainer" containerID="f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94" Nov 03 22:57:10.498616 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:10.498588 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94\": container with ID starting with f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94 not found: ID does not exist" containerID="f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94" Nov 03 22:57:10.498679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.498620 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94"} err="failed to get container status \"f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94\": rpc error: code = NotFound desc = could not find container \"f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94\": container with ID starting with f7f3ca14fe48d3fdfe9184f88a16ae20a8d4fea6cff3f59fde78a6b684548b94 not found: ID does not exist" Nov 03 22:57:10.498679 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.498643 5272 scope.go:122] "RemoveContainer" containerID="65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0" Nov 03 22:57:10.507937 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.507922 5272 scope.go:122] "RemoveContainer" containerID="65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0" Nov 03 22:57:10.508207 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:10.508188 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0\": container with ID starting with 65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0 not found: ID does not exist" containerID="65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0" Nov 03 22:57:10.508264 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.508216 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0"} err="failed to get container status \"65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0\": rpc error: code = NotFound desc = could not find container \"65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0\": container with ID starting with 65957aed926dc08c189c490723e4657a2fc1873bc483a7435ac6be103dafd7e0 not found: ID does not exist" Nov 03 22:57:10.716927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.714846 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j"] Nov 03 22:57:10.716927 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.716540 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0189ef-64d6f8fb8-r5l4j"] Nov 03 22:57:10.797483 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.797448 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb"] Nov 03 22:57:10.801245 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:10.801217 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0189ef-676855966c-2p5wb"] Nov 03 22:57:11.535002 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:11.534959 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2237d33f-a103-4ad7-908a-abaa7731658c" path="/var/lib/kubelet/pods/2237d33f-a103-4ad7-908a-abaa7731658c/volumes" Nov 03 22:57:11.535868 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:11.535828 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb5cb89-168d-4eac-863d-a769432088fe" path="/var/lib/kubelet/pods/6bb5cb89-168d-4eac-863d-a769432088fe/volumes" Nov 03 22:57:12.986710 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:12.986676 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b"] Nov 03 22:57:13.090836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:13.090791 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b"] Nov 03 22:57:17.987337 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:17.987300 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" podUID="50c2cc70-8d48-420b-8265-23ce3c630947" containerName="payload-processing" containerID="cri-o://ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8" gracePeriod=25 Nov 03 22:57:18.091996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.091926 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" podUID="9719dfa9-5c72-471f-8ded-b0f8b9031349" containerName="payload-pre-processing" containerID="cri-o://aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c" gracePeriod=25 Nov 03 22:57:18.272751 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.272721 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:57:18.338722 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.338678 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c2cc70-8d48-420b-8265-23ce3c630947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c2cc70-8d48-420b-8265-23ce3c630947-plugins-config-volume\") pod \"50c2cc70-8d48-420b-8265-23ce3c630947\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " Nov 03 22:57:18.338930 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.338760 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c2cc70-8d48-420b-8265-23ce3c630947-kube-api-access-c2lm7\" (UniqueName: \"kubernetes.io/projected/50c2cc70-8d48-420b-8265-23ce3c630947-kube-api-access-c2lm7\") pod \"50c2cc70-8d48-420b-8265-23ce3c630947\" (UID: \"50c2cc70-8d48-420b-8265-23ce3c630947\") " Nov 03 22:57:18.339140 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.339109 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c2cc70-8d48-420b-8265-23ce3c630947-plugins-config-volume" pod "50c2cc70-8d48-420b-8265-23ce3c630947" (UID: "50c2cc70-8d48-420b-8265-23ce3c630947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:18.339453 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.339433 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c2cc70-8d48-420b-8265-23ce3c630947-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:18.341886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.341846 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c2cc70-8d48-420b-8265-23ce3c630947-kube-api-access-c2lm7" pod "50c2cc70-8d48-420b-8265-23ce3c630947" (UID: "50c2cc70-8d48-420b-8265-23ce3c630947"). InnerVolumeSpecName "kube-api-access-c2lm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:18.370796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.370774 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:57:18.440429 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.440402 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9719dfa9-5c72-471f-8ded-b0f8b9031349-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9719dfa9-5c72-471f-8ded-b0f8b9031349-plugins-config-volume\") pod \"9719dfa9-5c72-471f-8ded-b0f8b9031349\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " Nov 03 22:57:18.440576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.440472 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9719dfa9-5c72-471f-8ded-b0f8b9031349-kube-api-access-5xgq6\" (UniqueName: \"kubernetes.io/projected/9719dfa9-5c72-471f-8ded-b0f8b9031349-kube-api-access-5xgq6\") pod \"9719dfa9-5c72-471f-8ded-b0f8b9031349\" (UID: \"9719dfa9-5c72-471f-8ded-b0f8b9031349\") " Nov 03 22:57:18.440782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.440748 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9719dfa9-5c72-471f-8ded-b0f8b9031349-plugins-config-volume" pod "9719dfa9-5c72-471f-8ded-b0f8b9031349" (UID: "9719dfa9-5c72-471f-8ded-b0f8b9031349"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:18.441014 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.440995 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9719dfa9-5c72-471f-8ded-b0f8b9031349-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:18.441086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.441020 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2lm7\" (UniqueName: \"kubernetes.io/projected/50c2cc70-8d48-420b-8265-23ce3c630947-kube-api-access-c2lm7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:18.442548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.442523 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9719dfa9-5c72-471f-8ded-b0f8b9031349-kube-api-access-5xgq6" pod "9719dfa9-5c72-471f-8ded-b0f8b9031349" (UID: "9719dfa9-5c72-471f-8ded-b0f8b9031349"). InnerVolumeSpecName "kube-api-access-5xgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:18.542475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.542334 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xgq6\" (UniqueName: \"kubernetes.io/projected/9719dfa9-5c72-471f-8ded-b0f8b9031349-kube-api-access-5xgq6\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:18.625065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.625034 5272 generic.go:356] "Generic (PLEG): container finished" podID="9719dfa9-5c72-471f-8ded-b0f8b9031349" containerID="aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c" exitCode=0 Nov 03 22:57:18.625249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.625077 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" Nov 03 22:57:18.625249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.625117 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" event={"ID":"9719dfa9-5c72-471f-8ded-b0f8b9031349","Type":"ContainerDied","Data":"aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c"} Nov 03 22:57:18.625249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.625170 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b" event={"ID":"9719dfa9-5c72-471f-8ded-b0f8b9031349","Type":"ContainerDied","Data":"b74dc6edff4f6b7b24b7875c3d901d6c6b0e923a9ccfaddedcb3673a7fb1a9e9"} Nov 03 22:57:18.625249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.625198 5272 scope.go:122] "RemoveContainer" containerID="aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c" Nov 03 22:57:18.626893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.626856 5272 generic.go:356] "Generic (PLEG): container finished" podID="50c2cc70-8d48-420b-8265-23ce3c630947" containerID="ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8" exitCode=0 Nov 03 22:57:18.627010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.626980 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" Nov 03 22:57:18.627010 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.626991 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" event={"ID":"50c2cc70-8d48-420b-8265-23ce3c630947","Type":"ContainerDied","Data":"ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8"} Nov 03 22:57:18.627126 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.627055 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b" event={"ID":"50c2cc70-8d48-420b-8265-23ce3c630947","Type":"ContainerDied","Data":"e12c2b8a9f45d309032cd87d6c839ff687ebc2c91310a383a717a5dd4215178a"} Nov 03 22:57:18.637612 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.637590 5272 scope.go:122] "RemoveContainer" containerID="aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c" Nov 03 22:57:18.637961 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:18.637940 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c\": container with ID starting with aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c not found: ID does not exist" containerID="aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c" Nov 03 22:57:18.638045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.637967 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c"} err="failed to get container status \"aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c\": rpc error: code = NotFound desc = could not find container \"aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c\": container with ID starting with aaa3d4e9607cae816a082e2e67470f722e9a638779291df17b59191ef152ff4c not found: ID does not exist" Nov 03 22:57:18.638045 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.637985 5272 scope.go:122] "RemoveContainer" containerID="ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8" Nov 03 22:57:18.651379 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.651354 5272 scope.go:122] "RemoveContainer" containerID="ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8" Nov 03 22:57:18.651767 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:18.651745 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8\": container with ID starting with ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8 not found: ID does not exist" containerID="ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8" Nov 03 22:57:18.651878 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.651775 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8"} err="failed to get container status \"ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8\": rpc error: code = NotFound desc = could not find container \"ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8\": container with ID starting with ca16a1599285c6c62034f847fd2b3fed20f7aada12c698c9fc16724021f5b4d8 not found: ID does not exist" Nov 03 22:57:18.780172 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.780134 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b"] Nov 03 22:57:18.784382 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.784356 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2420f6-b68c89c6-shx8b"] Nov 03 22:57:18.887139 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.887033 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b"] Nov 03 22:57:18.891256 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:18.891226 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2420f6-cb5bf5df5-9td4b"] Nov 03 22:57:19.495317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:19.495282 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c2cc70-8d48-420b-8265-23ce3c630947" path="/var/lib/kubelet/pods/50c2cc70-8d48-420b-8265-23ce3c630947/volumes" Nov 03 22:57:19.495703 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:19.495689 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9719dfa9-5c72-471f-8ded-b0f8b9031349" path="/var/lib/kubelet/pods/9719dfa9-5c72-471f-8ded-b0f8b9031349/volumes" Nov 03 22:57:24.501883 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:24.501841 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:57:25.603844 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.603794 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd"] Nov 03 22:57:25.604733 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604717 5272 cpu_manager.go:414] "containerMap: removing container" podUID="6bb5cb89-168d-4eac-863d-a769432088fe" containerName="payload-processing" Nov 03 22:57:25.604796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604737 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb5cb89-168d-4eac-863d-a769432088fe" containerName="payload-processing" Nov 03 22:57:25.604796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604769 5272 cpu_manager.go:414] "containerMap: removing container" podUID="69f67821-8bb2-49f4-9b23-f42bb1706448" containerName="revoke-keys" Nov 03 22:57:25.604796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604777 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f67821-8bb2-49f4-9b23-f42bb1706448" containerName="revoke-keys" Nov 03 22:57:25.604796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604791 5272 cpu_manager.go:414] "containerMap: removing container" podUID="176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" containerName="payload-processing" Nov 03 22:57:25.604796 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604813 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" containerName="payload-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604836 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d0152dff-65b4-433c-a22a-9e049aa15d41" containerName="payload-pre-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604844 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0152dff-65b4-433c-a22a-9e049aa15d41" containerName="payload-pre-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604861 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2237d33f-a103-4ad7-908a-abaa7731658c" containerName="payload-pre-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604869 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2237d33f-a103-4ad7-908a-abaa7731658c" containerName="payload-pre-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604901 5272 cpu_manager.go:414] "containerMap: removing container" podUID="50c2cc70-8d48-420b-8265-23ce3c630947" containerName="payload-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604908 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c2cc70-8d48-420b-8265-23ce3c630947" containerName="payload-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604920 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9719dfa9-5c72-471f-8ded-b0f8b9031349" containerName="payload-pre-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.604930 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9719dfa9-5c72-471f-8ded-b0f8b9031349" containerName="payload-pre-processing" Nov 03 22:57:25.605058 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605057 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f67821-8bb2-49f4-9b23-f42bb1706448" containerName="revoke-keys" Nov 03 22:57:25.605343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605068 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d448175a-3e0e-4b5d-b786-010d6a58a1af" containerName="istio-proxy" Nov 03 22:57:25.605343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605079 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="176f3cd2-99cf-4a3a-a2f7-db6c0da7df63" containerName="payload-processing" Nov 03 22:57:25.605343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605090 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0152dff-65b4-433c-a22a-9e049aa15d41" containerName="payload-pre-processing" Nov 03 22:57:25.605343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605100 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2237d33f-a103-4ad7-908a-abaa7731658c" containerName="payload-pre-processing" Nov 03 22:57:25.605343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605113 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c2cc70-8d48-420b-8265-23ce3c630947" containerName="payload-processing" Nov 03 22:57:25.605343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605124 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9719dfa9-5c72-471f-8ded-b0f8b9031349" containerName="payload-pre-processing" Nov 03 22:57:25.605343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.605135 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb5cb89-168d-4eac-863d-a769432088fe" containerName="payload-processing" Nov 03 22:57:25.610815 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.610782 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.711669 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.711628 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd"] Nov 03 22:57:25.743201 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743170 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743304 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743369 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743334 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57p2\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-kube-api-access-z57p2\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743451 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743375 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743451 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743420 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743487 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743509 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743594 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743529 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.743664 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.743624 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845394 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57p2\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-kube-api-access-z57p2\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845463 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845604 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845504 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845602 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845638 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845668 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845891 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845699 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.845891 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845746 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.846006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845910 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.846006 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.845962 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.846125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.846053 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.846125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.846088 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.846265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.846245 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.846652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.846633 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.847851 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.847827 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.848095 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.848076 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.903450 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.903387 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.903450 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.903398 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57p2\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-kube-api-access-z57p2\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-t67gd\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:25.925250 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:25.925225 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:26.374911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.374876 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd"] Nov 03 22:57:26.379350 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:57:26.379321 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a650cb_cc75_4d8e_b4bc_5772595e45bb.slice/crio-933a315ca2221ea5a7e306cd7616655b57f73026ff31117f2f6a677e501e790c WatchSource:0}: Error finding container 933a315ca2221ea5a7e306cd7616655b57f73026ff31117f2f6a677e501e790c: Status 404 returned error can't find the container with id 933a315ca2221ea5a7e306cd7616655b57f73026ff31117f2f6a677e501e790c Nov 03 22:57:26.382240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.382202 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:26.382317 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.382287 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:26.382360 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.382330 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:26.688972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.688869 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" event={"ID":"e5a650cb-cc75-4d8e-b4bc-5772595e45bb","Type":"ContainerStarted","Data":"fd416274e0158b093fd2b030053fdccf4714444d39ff98f42b02e95f208fe711"} Nov 03 22:57:26.688972 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.688930 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" event={"ID":"e5a650cb-cc75-4d8e-b4bc-5772595e45bb","Type":"ContainerStarted","Data":"933a315ca2221ea5a7e306cd7616655b57f73026ff31117f2f6a677e501e790c"} Nov 03 22:57:26.804389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.804336 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" podStartSLOduration=1.804322204 podStartE2EDuration="1.804322204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:26.80151709 +0000 UTC m=+2027.423825398" watchObservedRunningTime="2026-11-03 22:57:26.804322204 +0000 UTC m=+2027.426630512" Nov 03 22:57:26.925329 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:26.925288 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:27.928039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:27.928005 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:57:27.928557 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:27.928065 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:57:28.926548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:28.926515 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:57:28.926737 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:28.926579 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:57:29.926211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:29.926175 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:57:29.926688 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:29.926232 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:57:30.925996 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:30.925961 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:57:30.926188 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:30.926014 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:57:31.925696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:31.925661 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:57:31.926249 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:31.925726 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 03 22:57:32.930359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:32.930321 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:32.930772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:32.930753 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:32.931512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:32.931496 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 22:57:40.724516 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.724469 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc"] Nov 03 22:57:40.729207 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.729183 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.772316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.772276 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p"] Nov 03 22:57:40.772578 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.772546 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" containerID="cri-o://b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9" gracePeriod=30 Nov 03 22:57:40.787973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.787924 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600c464b-672f-46bd-8122-0b44bb476a9a-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788173 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.787979 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788248 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.788158 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600c464b-672f-46bd-8122-0b44bb476a9a-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788248 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.788203 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.788294 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788375 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.788340 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788559 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.788428 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4mv\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-kube-api-access-hb4mv\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.788625 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.788712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.788684 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.820828 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.819853 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc"] Nov 03 22:57:40.889795 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.889746 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.889862 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.889981 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600c464b-672f-46bd-8122-0b44bb476a9a-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890052 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890011 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890098 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600c464b-672f-46bd-8122-0b44bb476a9a-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890135 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890226 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890194 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890240 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890282 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4mv\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-kube-api-access-hb4mv\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890309 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-certs\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890467 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-credential-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890499 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-data\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890526 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-socket\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.890764 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.890737 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600c464b-672f-46bd-8122-0b44bb476a9a-istiod-ca-cert\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.892523 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.892499 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-envoy\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.892699 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.892678 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600c464b-672f-46bd-8122-0b44bb476a9a-istio-podinfo\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.947544 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.947502 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-istio-token\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:40.947712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:40.947590 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4mv\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-kube-api-access-hb4mv\") pod \"e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:41.043689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.043653 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:41.479366 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.479276 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc"] Nov 03 22:57:41.484130 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:57:41.484100 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600c464b_672f_46bd_8122_0b44bb476a9a.slice/crio-7c8730139912c2734b113ef0d9a8e9f7694bdabbddb6d1c3bd3f26592bfcd488 WatchSource:0}: Error finding container 7c8730139912c2734b113ef0d9a8e9f7694bdabbddb6d1c3bd3f26592bfcd488: Status 404 returned error can't find the container with id 7c8730139912c2734b113ef0d9a8e9f7694bdabbddb6d1c3bd3f26592bfcd488 Nov 03 22:57:41.486834 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.486786 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:41.486907 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.486878 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:41.486946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.486913 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:57:41.813685 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.813649 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" event={"ID":"600c464b-672f-46bd-8122-0b44bb476a9a","Type":"ContainerStarted","Data":"c35c0142876b35a0c5426c979892b44e66110233106dee9589df8c22ce58d2af"} Nov 03 22:57:41.814144 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.813707 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" event={"ID":"600c464b-672f-46bd-8122-0b44bb476a9a","Type":"ContainerStarted","Data":"7c8730139912c2734b113ef0d9a8e9f7694bdabbddb6d1c3bd3f26592bfcd488"} Nov 03 22:57:41.926904 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:41.926815 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podStartSLOduration=1.92677986 podStartE2EDuration="1.92677986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:41.926568574 +0000 UTC m=+2042.548876881" watchObservedRunningTime="2026-11-03 22:57:41.92677986 +0000 UTC m=+2042.549088165" Nov 03 22:57:42.044638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:42.044481 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:42.048920 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:42.048871 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:57:42.049246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:42.049211 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:57:42.841410 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:42.841373 5272 scope.go:122] "RemoveContainer" containerID="9e051dee4ff804fc5e2237d88f466b7ad6a7abe5a3a7ba2bfb47c83b5b412183" Nov 03 22:57:43.047234 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:43.047190 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:57:43.047521 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:43.047484 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:57:44.060826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:44.060163 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:57:44.060826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:44.060239 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:57:45.054185 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:45.054146 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:57:45.054485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:45.054454 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:57:46.068019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.051198 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:57:46.068019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.051262 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:57:46.089211 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.089183 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:57:46.188437 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188402 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-credential-socket\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.188622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188514 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-certs\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.188622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188574 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-podinfo\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.188622 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188619 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-envoy\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.188827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188670 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4574947-0021-4d1e-9c38-b781ee531c1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4574947-0021-4d1e-9c38-b781ee531c1b-istiod-ca-cert\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.188827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188711 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-data\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.188827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188749 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-socket\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.188827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188778 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-kube-api-access-6bk6l\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-kube-api-access-6bk6l\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.189062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188835 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-token\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-token\") pod \"b4574947-0021-4d1e-9c38-b781ee531c1b\" (UID: \"b4574947-0021-4d1e-9c38-b781ee531c1b\") " Nov 03 22:57:46.189062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188616 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-credential-socket" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:46.189062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188821 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-certs" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:46.189062 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.188976 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-data" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:46.189266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.189129 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-socket" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:46.189266 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.189169 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4574947-0021-4d1e-9c38-b781ee531c1b-istiod-ca-cert" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:46.189652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.189553 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4574947-0021-4d1e-9c38-b781ee531c1b-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.189652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.189579 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.189652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.189594 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.189652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.189609 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.189652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.189623 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.191034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.191010 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-envoy" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:46.191441 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.191421 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-podinfo" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:57:46.191776 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.191745 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-token" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:46.191917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.191894 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-kube-api-access-6bk6l" pod "b4574947-0021-4d1e-9c38-b781ee531c1b" (UID: "b4574947-0021-4d1e-9c38-b781ee531c1b"). InnerVolumeSpecName "kube-api-access-6bk6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:46.291739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.291646 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.291739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.291687 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.291739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.291702 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bk6l\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-kube-api-access-6bk6l\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.291739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.291718 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4574947-0021-4d1e-9c38-b781ee531c1b-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:57:46.876489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.876451 5272 generic.go:356] "Generic (PLEG): container finished" podID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerID="b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9" exitCode=0 Nov 03 22:57:46.876682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.876516 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" Nov 03 22:57:46.876682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.876535 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" event={"ID":"b4574947-0021-4d1e-9c38-b781ee531c1b","Type":"ContainerDied","Data":"b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9"} Nov 03 22:57:46.876682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.876623 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p" event={"ID":"b4574947-0021-4d1e-9c38-b781ee531c1b","Type":"ContainerDied","Data":"a71dfcdae93fc2f48039a1fd777776e597ee8cc4d64dbfe71f5bd230d8f3f64e"} Nov 03 22:57:46.876682 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.876648 5272 scope.go:122] "RemoveContainer" containerID="b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9" Nov 03 22:57:46.888338 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.888318 5272 scope.go:122] "RemoveContainer" containerID="b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9" Nov 03 22:57:46.888618 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:46.888600 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9\": container with ID starting with b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9 not found: ID does not exist" containerID="b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9" Nov 03 22:57:46.888695 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:46.888626 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9"} err="failed to get container status \"b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9\": rpc error: code = NotFound desc = could not find container \"b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9\": container with ID starting with b052797a611d9aefe242a347d0843c4285b2450ca103c6e81f8887f4873ca2a9 not found: ID does not exist" Nov 03 22:57:47.033914 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:47.033877 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p"] Nov 03 22:57:47.065827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:47.053166 5272 patch_prober.go:29] interesting pod/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 22:57:47.065827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:47.053227 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 22:57:47.079862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:47.079830 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-wfg5p"] Nov 03 22:57:47.494591 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:47.494557 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" path="/var/lib/kubelet/pods/b4574947-0021-4d1e-9c38-b781ee531c1b/volumes" Nov 03 22:57:48.049176 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:48.049147 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:48.049628 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:48.049608 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:48.050389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:57:48.050372 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 22:57:54.501719 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:57:54.501677 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:58:02.046696 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:02.046642 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn"] Nov 03 22:58:02.092987 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:02.092957 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e656460-52gqn"] Nov 03 22:58:03.524871 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:03.520712 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2551c891-97ad-4d84-87ce-5038bd9c2680" path="/var/lib/kubelet/pods/2551c891-97ad-4d84-87ce-5038bd9c2680/volumes" Nov 03 22:58:04.690469 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:04.690435 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6"] Nov 03 22:58:04.737118 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:04.737090 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-65913e-jq8h6"] Nov 03 22:58:05.500453 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:05.500420 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069c76bb-7918-40f1-aaef-f4a134527013" path="/var/lib/kubelet/pods/069c76bb-7918-40f1-aaef-f4a134527013/volumes" Nov 03 22:58:12.364773 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.364737 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48"] Nov 03 22:58:12.365579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.365560 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" Nov 03 22:58:12.365579 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.365580 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" Nov 03 22:58:12.365735 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.365724 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4574947-0021-4d1e-9c38-b781ee531c1b" containerName="istio-proxy" Nov 03 22:58:12.369547 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.369531 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.419393 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.419371 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:58:12.439053 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.439024 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.439175 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.439089 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.439306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.439279 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24vd\" (UniqueName: \"kubernetes.io/projected/4e70e407-d8fe-4600-842c-68e150e71414-kube-api-access-h24vd\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.465783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.465755 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:12.465783 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.465774 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:58:12.465991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.465834 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:58:12.480097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.480069 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48"] Nov 03 22:58:12.540370 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.540336 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.540532 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.540460 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.540674 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.540590 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24vd\" (UniqueName: \"kubernetes.io/projected/4e70e407-d8fe-4600-842c-68e150e71414-kube-api-access-h24vd\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.541265 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.541244 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.543137 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.543111 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.599533 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.599503 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24vd\" (UniqueName: \"kubernetes.io/projected/4e70e407-d8fe-4600-842c-68e150e71414-kube-api-access-h24vd\") pod \"maas-api-revoke-keys-e2e-mt-12450a06-dkw48\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:12.683447 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:12.683384 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:13.109908 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:13.109718 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48"] Nov 03 22:58:13.114900 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:58:13.114868 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e70e407_d8fe_4600_842c_68e150e71414.slice/crio-042c4f1f9b39a00ac9cb4cdd11efec3be9d755bf2cf9a1b1d5690fa311f03a67 WatchSource:0}: Error finding container 042c4f1f9b39a00ac9cb4cdd11efec3be9d755bf2cf9a1b1d5690fa311f03a67: Status 404 returned error can't find the container with id 042c4f1f9b39a00ac9cb4cdd11efec3be9d755bf2cf9a1b1d5690fa311f03a67 Nov 03 22:58:14.092894 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:14.092854 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" event={"ID":"4e70e407-d8fe-4600-842c-68e150e71414","Type":"ContainerStarted","Data":"9a8f307ecc08e74f8a1be20afe58daf1ba7f44046c9889c7ab4658010309e9e6"} Nov 03 22:58:14.093307 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:14.092915 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" event={"ID":"4e70e407-d8fe-4600-842c-68e150e71414","Type":"ContainerStarted","Data":"042c4f1f9b39a00ac9cb4cdd11efec3be9d755bf2cf9a1b1d5690fa311f03a67"} Nov 03 22:58:14.219958 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:14.219898 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" podStartSLOduration=2.219881753 podStartE2EDuration="2.219881753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:14.201540181 +0000 UTC m=+2074.823848489" watchObservedRunningTime="2026-11-03 22:58:14.219881753 +0000 UTC m=+2074.842190043" Nov 03 22:58:15.102653 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:15.102622 5272 generic.go:356] "Generic (PLEG): container finished" podID="4e70e407-d8fe-4600-842c-68e150e71414" containerID="9a8f307ecc08e74f8a1be20afe58daf1ba7f44046c9889c7ab4658010309e9e6" exitCode=0 Nov 03 22:58:15.103125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:15.102658 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" event={"ID":"4e70e407-d8fe-4600-842c-68e150e71414","Type":"ContainerDied","Data":"9a8f307ecc08e74f8a1be20afe58daf1ba7f44046c9889c7ab4658010309e9e6"} Nov 03 22:58:16.274600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.274572 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:16.295254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.295222 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e70e407-d8fe-4600-842c-68e150e71414-kube-api-access-h24vd\" (UniqueName: \"kubernetes.io/projected/4e70e407-d8fe-4600-842c-68e150e71414-kube-api-access-h24vd\") pod \"4e70e407-d8fe-4600-842c-68e150e71414\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " Nov 03 22:58:16.295412 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.295397 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-xks-ca\") pod \"4e70e407-d8fe-4600-842c-68e150e71414\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " Nov 03 22:58:16.295467 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.295441 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-openshift-ca\") pod \"4e70e407-d8fe-4600-842c-68e150e71414\" (UID: \"4e70e407-d8fe-4600-842c-68e150e71414\") " Nov 03 22:58:16.295875 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.295848 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-openshift-ca" pod "4e70e407-d8fe-4600-842c-68e150e71414" (UID: "4e70e407-d8fe-4600-842c-68e150e71414"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:16.295994 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.295972 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:16.297454 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.297431 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e70e407-d8fe-4600-842c-68e150e71414-kube-api-access-h24vd" pod "4e70e407-d8fe-4600-842c-68e150e71414" (UID: "4e70e407-d8fe-4600-842c-68e150e71414"). InnerVolumeSpecName "kube-api-access-h24vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:16.297553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.297452 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-xks-ca" pod "4e70e407-d8fe-4600-842c-68e150e71414" (UID: "4e70e407-d8fe-4600-842c-68e150e71414"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:16.397595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.397509 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e70e407-d8fe-4600-842c-68e150e71414-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:16.397595 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:16.397553 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h24vd\" (UniqueName: \"kubernetes.io/projected/4e70e407-d8fe-4600-842c-68e150e71414-kube-api-access-h24vd\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:17.121331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:17.121289 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" Nov 03 22:58:17.121331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:17.121308 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48" event={"ID":"4e70e407-d8fe-4600-842c-68e150e71414","Type":"ContainerDied","Data":"042c4f1f9b39a00ac9cb4cdd11efec3be9d755bf2cf9a1b1d5690fa311f03a67"} Nov 03 22:58:17.121592 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:17.121368 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="042c4f1f9b39a00ac9cb4cdd11efec3be9d755bf2cf9a1b1d5690fa311f03a67" Nov 03 22:58:21.726030 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:21.725956 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8"] Nov 03 22:58:21.772790 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:21.772762 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0189ef-sr6l8"] Nov 03 22:58:23.494212 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:23.494176 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62213f58-aaf7-4526-be69-8209a46b7212" path="/var/lib/kubelet/pods/62213f58-aaf7-4526-be69-8209a46b7212/volumes" Nov 03 22:58:24.509406 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:58:24.509366 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:58:26.311586 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.311552 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn"] Nov 03 22:58:26.312303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.312273 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4e70e407-d8fe-4600-842c-68e150e71414" containerName="revoke-keys" Nov 03 22:58:26.312303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.312299 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e70e407-d8fe-4600-842c-68e150e71414" containerName="revoke-keys" Nov 03 22:58:26.312464 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.312426 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e70e407-d8fe-4600-842c-68e150e71414" containerName="revoke-keys" Nov 03 22:58:26.316103 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.316085 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.368949 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.368922 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn"] Nov 03 22:58:26.411233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.411210 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:26.411400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.411266 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:58:26.411400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.411278 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:58:26.411400 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.411286 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:58:26.460279 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.460255 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfx9\" (UniqueName: \"kubernetes.io/projected/c6b9b2ec-7256-424e-82e3-29e6288800d6-kube-api-access-8jfx9\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.460439 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.460419 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.460564 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.460546 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.561352 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.561319 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.561536 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.561416 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.561536 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.561501 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfx9\" (UniqueName: \"kubernetes.io/projected/c6b9b2ec-7256-424e-82e3-29e6288800d6-kube-api-access-8jfx9\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.562031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.561973 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.563777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.563754 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.618792 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.618760 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfx9\" (UniqueName: \"kubernetes.io/projected/c6b9b2ec-7256-424e-82e3-29e6288800d6-kube-api-access-8jfx9\") pod \"maas-api-revoke-sub-6284123f-zp9xn\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:26.629548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:26.629510 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:27.025899 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:27.025789 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn"] Nov 03 22:58:27.030985 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:58:27.030953 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b9b2ec_7256_424e_82e3_29e6288800d6.slice/crio-5c71b4dd5a886e0b8f08b7e42679900c45c9263214340528328ede3a6ac95d9b WatchSource:0}: Error finding container 5c71b4dd5a886e0b8f08b7e42679900c45c9263214340528328ede3a6ac95d9b: Status 404 returned error can't find the container with id 5c71b4dd5a886e0b8f08b7e42679900c45c9263214340528328ede3a6ac95d9b Nov 03 22:58:27.193617 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:27.193586 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" event={"ID":"c6b9b2ec-7256-424e-82e3-29e6288800d6","Type":"ContainerStarted","Data":"5c71b4dd5a886e0b8f08b7e42679900c45c9263214340528328ede3a6ac95d9b"} Nov 03 22:58:28.203730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:28.203632 5272 generic.go:356] "Generic (PLEG): container finished" podID="c6b9b2ec-7256-424e-82e3-29e6288800d6" containerID="8241c7fa9675122986f38dfc8d7d17a035858c0a3c8e7e2a5202475e87822206" exitCode=0 Nov 03 22:58:28.204156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:28.203726 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" event={"ID":"c6b9b2ec-7256-424e-82e3-29e6288800d6","Type":"ContainerDied","Data":"8241c7fa9675122986f38dfc8d7d17a035858c0a3c8e7e2a5202475e87822206"} Nov 03 22:58:29.398404 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.398380 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:29.498129 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.498042 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-xks-ca\") pod \"c6b9b2ec-7256-424e-82e3-29e6288800d6\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " Nov 03 22:58:29.498310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.498179 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b9b2ec-7256-424e-82e3-29e6288800d6-kube-api-access-8jfx9\" (UniqueName: \"kubernetes.io/projected/c6b9b2ec-7256-424e-82e3-29e6288800d6-kube-api-access-8jfx9\") pod \"c6b9b2ec-7256-424e-82e3-29e6288800d6\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " Nov 03 22:58:29.498310 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.498244 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-openshift-ca\") pod \"c6b9b2ec-7256-424e-82e3-29e6288800d6\" (UID: \"c6b9b2ec-7256-424e-82e3-29e6288800d6\") " Nov 03 22:58:29.499384 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.499351 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-openshift-ca" pod "c6b9b2ec-7256-424e-82e3-29e6288800d6" (UID: "c6b9b2ec-7256-424e-82e3-29e6288800d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:29.503782 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.503752 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-xks-ca" pod "c6b9b2ec-7256-424e-82e3-29e6288800d6" (UID: "c6b9b2ec-7256-424e-82e3-29e6288800d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:29.504028 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.504009 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b9b2ec-7256-424e-82e3-29e6288800d6-kube-api-access-8jfx9" pod "c6b9b2ec-7256-424e-82e3-29e6288800d6" (UID: "c6b9b2ec-7256-424e-82e3-29e6288800d6"). InnerVolumeSpecName "kube-api-access-8jfx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:29.601830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.600938 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:29.601830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.600971 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b9b2ec-7256-424e-82e3-29e6288800d6-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:29.601830 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:29.600986 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfx9\" (UniqueName: \"kubernetes.io/projected/c6b9b2ec-7256-424e-82e3-29e6288800d6-kube-api-access-8jfx9\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.232986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.232948 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" event={"ID":"c6b9b2ec-7256-424e-82e3-29e6288800d6","Type":"ContainerDied","Data":"5c71b4dd5a886e0b8f08b7e42679900c45c9263214340528328ede3a6ac95d9b"} Nov 03 22:58:30.232986 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.232971 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn" Nov 03 22:58:30.233198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.233006 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c71b4dd5a886e0b8f08b7e42679900c45c9263214340528328ede3a6ac95d9b" Nov 03 22:58:30.474512 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.474477 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:58:30.478475 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.478452 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2"] Nov 03 22:58:30.479149 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.479124 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c6b9b2ec-7256-424e-82e3-29e6288800d6" containerName="revoke-keys" Nov 03 22:58:30.479149 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.479143 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b9b2ec-7256-424e-82e3-29e6288800d6" containerName="revoke-keys" Nov 03 22:58:30.479351 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.479275 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b9b2ec-7256-424e-82e3-29e6288800d6" containerName="revoke-keys" Nov 03 22:58:30.483056 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.483005 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.545833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.545052 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-31ad7b-openshift-default-dockercfg-ts5jl\"" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.698680 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-socket\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.698825 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9060091-788b-47ed-8934-18fc8c9bc462-istio-podinfo\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.698936 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-istio-token\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.699019 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9060091-788b-47ed-8934-18fc8c9bc462-istiod-ca-cert\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.699098 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-data\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.699208 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-certs\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.699241 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-envoy\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.699284 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-credential-socket\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.699335 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lslb\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-kube-api-access-7lslb\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.726829 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.715221 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2"] Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805470 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-istio-token\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805550 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9060091-788b-47ed-8934-18fc8c9bc462-istiod-ca-cert\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805628 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-data\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805715 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-certs\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805746 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-envoy\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805786 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-credential-socket\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805849 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lslb\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-kube-api-access-7lslb\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805878 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-socket\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.805975 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9060091-788b-47ed-8934-18fc8c9bc462-istio-podinfo\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.806385 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-certs\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.806854 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.806512 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-credential-socket\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.807485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.806950 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-data\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.807485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.807040 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-socket\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.807485 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.807100 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9060091-788b-47ed-8934-18fc8c9bc462-istiod-ca-cert\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.810671 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.810645 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-envoy\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.813242 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.813218 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9060091-788b-47ed-8934-18fc8c9bc462-istio-podinfo\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.875575 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.875540 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lslb\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-kube-api-access-7lslb\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:30.883111 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:30.883080 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-istio-token\") pod \"e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:31.098166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:31.098070 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:31.647893 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:31.647289 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2"] Nov 03 22:58:31.686312 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:58:31.686268 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9060091_788b_47ed_8934_18fc8c9bc462.slice/crio-36ca01dd93ca0c9438ca48d34f9248144b225bafeb06f093da1d819a4c68b635 WatchSource:0}: Error finding container 36ca01dd93ca0c9438ca48d34f9248144b225bafeb06f093da1d819a4c68b635: Status 404 returned error can't find the container with id 36ca01dd93ca0c9438ca48d34f9248144b225bafeb06f093da1d819a4c68b635 Nov 03 22:58:31.697833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:31.690281 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:58:31.697833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:31.690395 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:58:31.697833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:31.690440 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:58:32.331655 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:32.331614 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" event={"ID":"f9060091-788b-47ed-8934-18fc8c9bc462","Type":"ContainerStarted","Data":"cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c"} Nov 03 22:58:32.331884 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:32.331673 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" event={"ID":"f9060091-788b-47ed-8934-18fc8c9bc462","Type":"ContainerStarted","Data":"36ca01dd93ca0c9438ca48d34f9248144b225bafeb06f093da1d819a4c68b635"} Nov 03 22:58:32.445946 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:32.445898 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podStartSLOduration=2.44588519 podStartE2EDuration="2.44588519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:32.444364348 +0000 UTC m=+2093.066672656" watchObservedRunningTime="2026-11-03 22:58:32.44588519 +0000 UTC m=+2093.068193497" Nov 03 22:58:32.933304 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:32.933268 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6"] Nov 03 22:58:32.941738 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:32.941695 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c21c7-4p2p6"] Nov 03 22:58:33.099882 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:33.099844 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:33.105563 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:33.105534 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:33.338747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:33.338712 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:33.339672 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:33.339651 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 22:58:33.498069 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:33.498027 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d553e1-48aa-4209-8724-656856b00bbf" path="/var/lib/kubelet/pods/f1d553e1-48aa-4209-8724-656856b00bbf/volumes" Nov 03 22:58:38.869685 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:38.869651 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh"] Nov 03 22:58:38.870109 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:38.869956 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" containerID="cri-o://e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb" gracePeriod=30 Nov 03 22:58:39.662490 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:39.662450 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99"] Nov 03 22:58:39.665433 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:39.665407 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2420f6-9jv99"] Nov 03 22:58:40.107316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.107276 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx"] Nov 03 22:58:40.113239 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.113214 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.162652 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.162600 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:40.164790 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.164762 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx"] Nov 03 22:58:40.206615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.206585 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:58:40.206826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.206717 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:58:40.206826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.206780 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:58:40.229570 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.229526 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.229739 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.229671 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.229862 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.229794 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7s72\" (UniqueName: \"kubernetes.io/projected/69d5eb43-1d67-4bba-b9f3-e347d9176d98-kube-api-access-n7s72\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.331727 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.331692 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7s72\" (UniqueName: \"kubernetes.io/projected/69d5eb43-1d67-4bba-b9f3-e347d9176d98-kube-api-access-n7s72\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.331941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.331846 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.331941 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.331914 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.332636 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.332609 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.334380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.334347 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.389303 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.389228 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7s72\" (UniqueName: \"kubernetes.io/projected/69d5eb43-1d67-4bba-b9f3-e347d9176d98-kube-api-access-n7s72\") pod \"maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.430815 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.430766 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:40.820533 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:40.820501 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx"] Nov 03 22:58:40.825978 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:58:40.825937 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d5eb43_1d67_4bba_b9f3_e347d9176d98.slice/crio-47f43b7d4a7924305a51f85ea2a5fc16ede4d797831ea86a879c07710747a314 WatchSource:0}: Error finding container 47f43b7d4a7924305a51f85ea2a5fc16ede4d797831ea86a879c07710747a314: Status 404 returned error can't find the container with id 47f43b7d4a7924305a51f85ea2a5fc16ede4d797831ea86a879c07710747a314 Nov 03 22:58:41.444584 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:41.444486 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" event={"ID":"69d5eb43-1d67-4bba-b9f3-e347d9176d98","Type":"ContainerStarted","Data":"f2019c03e28d35a08f51dfb2bf5f31312cf15e06eea2ce2631dbb4cdb49b12ab"} Nov 03 22:58:41.444584 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:41.444559 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" event={"ID":"69d5eb43-1d67-4bba-b9f3-e347d9176d98","Type":"ContainerStarted","Data":"47f43b7d4a7924305a51f85ea2a5fc16ede4d797831ea86a879c07710747a314"} Nov 03 22:58:41.494513 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:41.494476 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f67821-8bb2-49f4-9b23-f42bb1706448" path="/var/lib/kubelet/pods/69f67821-8bb2-49f4-9b23-f42bb1706448/volumes" Nov 03 22:58:41.557917 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:41.557857 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" podStartSLOduration=1.557838747 podStartE2EDuration="1.557838747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:41.556662756 +0000 UTC m=+2102.178971066" watchObservedRunningTime="2026-11-03 22:58:41.557838747 +0000 UTC m=+2102.180147059" Nov 03 22:58:42.452928 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.452889 5272 generic.go:356] "Generic (PLEG): container finished" podID="69d5eb43-1d67-4bba-b9f3-e347d9176d98" containerID="f2019c03e28d35a08f51dfb2bf5f31312cf15e06eea2ce2631dbb4cdb49b12ab" exitCode=0 Nov 03 22:58:42.453422 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.452961 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" event={"ID":"69d5eb43-1d67-4bba-b9f3-e347d9176d98","Type":"ContainerDied","Data":"f2019c03e28d35a08f51dfb2bf5f31312cf15e06eea2ce2631dbb4cdb49b12ab"} Nov 03 22:58:42.454879 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.454858 5272 generic.go:356] "Generic (PLEG): container finished" podID="c27d9423-3e14-44de-a112-3edc286ab845" containerID="409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1" exitCode=0 Nov 03 22:58:42.455015 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.454903 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" event={"ID":"c27d9423-3e14-44de-a112-3edc286ab845","Type":"ContainerDied","Data":"409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1"} Nov 03 22:58:42.455306 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.455289 5272 scope.go:122] "RemoveContainer" containerID="409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1" Nov 03 22:58:42.456886 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.456858 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:58:42.456953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.456925 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:58:42.456991 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.456955 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:58:42.942097 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:42.942067 5272 scope.go:122] "RemoveContainer" containerID="ed7167abc2a5d51858d4a9df94b165739fd537e213b55cd02edd1ce5b7801015" Nov 03 22:58:43.471568 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.471527 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" event={"ID":"c27d9423-3e14-44de-a112-3edc286ab845","Type":"ContainerStarted","Data":"8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e"} Nov 03 22:58:43.680391 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.680075 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:43.713040 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.713009 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:58:43.714026 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.713999 5272 patch_prober.go:29] interesting pod/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 22:58:43.714134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.714047 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 22:58:43.778419 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.778323 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-xks-ca\") pod \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " Nov 03 22:58:43.778419 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.778363 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d5eb43-1d67-4bba-b9f3-e347d9176d98-kube-api-access-n7s72\" (UniqueName: \"kubernetes.io/projected/69d5eb43-1d67-4bba-b9f3-e347d9176d98-kube-api-access-n7s72\") pod \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " Nov 03 22:58:43.778419 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.778412 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-openshift-ca\") pod \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\" (UID: \"69d5eb43-1d67-4bba-b9f3-e347d9176d98\") " Nov 03 22:58:43.778861 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.778829 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-openshift-ca" pod "69d5eb43-1d67-4bba-b9f3-e347d9176d98" (UID: "69d5eb43-1d67-4bba-b9f3-e347d9176d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:43.779206 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.779188 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:43.780543 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.780519 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-xks-ca" pod "69d5eb43-1d67-4bba-b9f3-e347d9176d98" (UID: "69d5eb43-1d67-4bba-b9f3-e347d9176d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:43.780643 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.780595 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d5eb43-1d67-4bba-b9f3-e347d9176d98-kube-api-access-n7s72" pod "69d5eb43-1d67-4bba-b9f3-e347d9176d98" (UID: "69d5eb43-1d67-4bba-b9f3-e347d9176d98"). InnerVolumeSpecName "kube-api-access-n7s72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:43.880359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.880320 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69d5eb43-1d67-4bba-b9f3-e347d9176d98-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:43.880359 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:43.880360 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7s72\" (UniqueName: \"kubernetes.io/projected/69d5eb43-1d67-4bba-b9f3-e347d9176d98-kube-api-access-n7s72\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.153383 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.153016 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:58:44.184610 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.184575 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-certs\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.184842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.184672 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-kube-api-access-zdhbk\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-kube-api-access-zdhbk\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.184842 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.184718 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-credential-socket\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.184997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.184913 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-socket\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.184997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.184927 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-certs" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:44.185108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185040 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45585d1a-823c-4006-bb93-f257c3dcb895-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45585d1a-823c-4006-bb93-f257c3dcb895-istiod-ca-cert\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.185108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185051 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-credential-socket" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:44.185108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185089 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-data\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.185294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185147 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45585d1a-823c-4006-bb93-f257c3dcb895-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45585d1a-823c-4006-bb93-f257c3dcb895-istio-podinfo\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.185294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185160 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-socket" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:44.185294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185230 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-istio-token\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-istio-token\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.185294 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185287 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-envoy\") pod \"45585d1a-823c-4006-bb93-f257c3dcb895\" (UID: \"45585d1a-823c-4006-bb93-f257c3dcb895\") " Nov 03 22:58:44.185520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185344 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-data" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:44.185520 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185425 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45585d1a-823c-4006-bb93-f257c3dcb895-istiod-ca-cert" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:44.185853 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185795 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.185982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185856 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.185982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185870 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.185982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185885 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45585d1a-823c-4006-bb93-f257c3dcb895-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.185982 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.185900 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.187134 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.187114 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45585d1a-823c-4006-bb93-f257c3dcb895-istio-podinfo" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:44.187339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.187310 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-kube-api-access-zdhbk" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "kube-api-access-zdhbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:44.187452 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.187373 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-envoy" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:44.187547 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.187524 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-istio-token" pod "45585d1a-823c-4006-bb93-f257c3dcb895" (UID: "45585d1a-823c-4006-bb93-f257c3dcb895"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:44.287554 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.287514 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdhbk\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-kube-api-access-zdhbk\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.287554 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.287553 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45585d1a-823c-4006-bb93-f257c3dcb895-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.287774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.287568 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45585d1a-823c-4006-bb93-f257c3dcb895-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.287774 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.287583 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45585d1a-823c-4006-bb93-f257c3dcb895-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:44.480397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.480316 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" Nov 03 22:58:44.480397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.480341 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx" event={"ID":"69d5eb43-1d67-4bba-b9f3-e347d9176d98","Type":"ContainerDied","Data":"47f43b7d4a7924305a51f85ea2a5fc16ede4d797831ea86a879c07710747a314"} Nov 03 22:58:44.480397 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.480391 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47f43b7d4a7924305a51f85ea2a5fc16ede4d797831ea86a879c07710747a314" Nov 03 22:58:44.482331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.482304 5272 generic.go:356] "Generic (PLEG): container finished" podID="45585d1a-823c-4006-bb93-f257c3dcb895" containerID="e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb" exitCode=0 Nov 03 22:58:44.482461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.482351 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" Nov 03 22:58:44.482461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.482387 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" event={"ID":"45585d1a-823c-4006-bb93-f257c3dcb895","Type":"ContainerDied","Data":"e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb"} Nov 03 22:58:44.482461 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.482444 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh" event={"ID":"45585d1a-823c-4006-bb93-f257c3dcb895","Type":"ContainerDied","Data":"201ffe147f82fbffd5763a8bf776035e0f6f170b0cdb17cc2527b0f56c579cd2"} Nov 03 22:58:44.482571 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.482475 5272 scope.go:122] "RemoveContainer" containerID="e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb" Nov 03 22:58:44.493749 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.493732 5272 scope.go:122] "RemoveContainer" containerID="9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf" Nov 03 22:58:44.502922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.502904 5272 scope.go:122] "RemoveContainer" containerID="e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb" Nov 03 22:58:44.503164 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:58:44.503144 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb\": container with ID starting with e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb not found: ID does not exist" containerID="e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb" Nov 03 22:58:44.503217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.503170 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb"} err="failed to get container status \"e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb\": rpc error: code = NotFound desc = could not find container \"e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb\": container with ID starting with e666a5a8b27c29a2296f6501a4c914c932e10ef55ae9aada229944379b73d6eb not found: ID does not exist" Nov 03 22:58:44.503217 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.503188 5272 scope.go:122] "RemoveContainer" containerID="9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf" Nov 03 22:58:44.503410 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:58:44.503391 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf\": container with ID starting with 9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf not found: ID does not exist" containerID="9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf" Nov 03 22:58:44.503457 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.503416 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf"} err="failed to get container status \"9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf\": rpc error: code = NotFound desc = could not find container \"9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf\": container with ID starting with 9353ac4277a4eb82bb04c085ef39341aa4a41cd1330e2bf517e406b55c5f24cf not found: ID does not exist" Nov 03 22:58:44.686985 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.686932 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh"] Nov 03 22:58:44.690673 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.690646 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0c21c7-openshift-default-8476d54f65-ph5gh"] Nov 03 22:58:44.712531 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.712506 5272 patch_prober.go:29] interesting pod/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 22:58:44.712642 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:44.712561 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 22:58:45.040912 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:45.040878 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:58:45.494321 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:45.494241 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" path="/var/lib/kubelet/pods/45585d1a-823c-4006-bb93-f257c3dcb895/volumes" Nov 03 22:58:45.713140 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:45.713100 5272 patch_prober.go:29] interesting pod/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 22:58:45.713344 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:45.713171 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 22:58:46.713314 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:46.713277 5272 patch_prober.go:29] interesting pod/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 22:58:46.713714 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:46.713338 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 22:58:47.500034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.499999 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb"] Nov 03 22:58:47.500689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500672 5272 cpu_manager.go:414] "containerMap: removing container" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" Nov 03 22:58:47.500763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500690 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" Nov 03 22:58:47.500763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500705 5272 cpu_manager.go:414] "containerMap: removing container" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" Nov 03 22:58:47.500763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500747 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" Nov 03 22:58:47.500763 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500762 5272 cpu_manager.go:414] "containerMap: removing container" podUID="69d5eb43-1d67-4bba-b9f3-e347d9176d98" containerName="revoke-keys" Nov 03 22:58:47.500993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500768 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d5eb43-1d67-4bba-b9f3-e347d9176d98" containerName="revoke-keys" Nov 03 22:58:47.500993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500925 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" Nov 03 22:58:47.500993 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.500938 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d5eb43-1d67-4bba-b9f3-e347d9176d98" containerName="revoke-keys" Nov 03 22:58:47.505055 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.505036 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.570827 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.569972 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:47.571824 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.571644 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb"] Nov 03 22:58:47.593538 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.593506 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.593689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.593574 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.593689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.593642 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/76a503b3-b09b-48d7-a396-2afe6dab2f0d-kube-api-access-xmb22\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.609647 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.609619 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:58:47.609852 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.609799 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:58:47.610026 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.609686 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:58:47.695158 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.695123 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.695340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.695188 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.695340 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.695211 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/76a503b3-b09b-48d7-a396-2afe6dab2f0d-kube-api-access-xmb22\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.695903 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.695879 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.697530 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.697507 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.712576 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.712551 5272 patch_prober.go:29] interesting pod/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 22:58:47.712704 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.712603 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 22:58:47.768553 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.768466 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/76a503b3-b09b-48d7-a396-2afe6dab2f0d-kube-api-access-xmb22\") pod \"maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:47.818406 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:47.818378 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:48.222918 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:48.222882 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb"] Nov 03 22:58:48.228726 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:58:48.228693 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a503b3_b09b_48d7_a396_2afe6dab2f0d.slice/crio-3309cd4f6a91c2604517add648191bd9eb5b666e0d4ba73d70980cf672af81a9 WatchSource:0}: Error finding container 3309cd4f6a91c2604517add648191bd9eb5b666e0d4ba73d70980cf672af81a9: Status 404 returned error can't find the container with id 3309cd4f6a91c2604517add648191bd9eb5b666e0d4ba73d70980cf672af81a9 Nov 03 22:58:48.514380 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:48.514346 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" event={"ID":"76a503b3-b09b-48d7-a396-2afe6dab2f0d","Type":"ContainerStarted","Data":"3309cd4f6a91c2604517add648191bd9eb5b666e0d4ba73d70980cf672af81a9"} Nov 03 22:58:48.712685 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:48.712647 5272 patch_prober.go:29] interesting pod/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Nov 03 22:58:48.712897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:48.712710 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Nov 03 22:58:49.523371 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:49.523346 5272 generic.go:356] "Generic (PLEG): container finished" podID="76a503b3-b09b-48d7-a396-2afe6dab2f0d" containerID="dcbad510245848ec1a09efbb4f50e26383f6e729bea8867d1c34f74f56874a6c" exitCode=0 Nov 03 22:58:49.523702 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:49.523434 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" event={"ID":"76a503b3-b09b-48d7-a396-2afe6dab2f0d","Type":"ContainerDied","Data":"dcbad510245848ec1a09efbb4f50e26383f6e729bea8867d1c34f74f56874a6c"} Nov 03 22:58:49.717161 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:49.717134 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:58:49.718284 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:49.718264 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 22:58:50.684327 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.684301 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:50.840065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.840034 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-openshift-ca\") pod \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " Nov 03 22:58:50.840213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.840118 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-xks-ca\") pod \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " Nov 03 22:58:50.840213 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.840160 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a503b3-b09b-48d7-a396-2afe6dab2f0d-kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/76a503b3-b09b-48d7-a396-2afe6dab2f0d-kube-api-access-xmb22\") pod \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\" (UID: \"76a503b3-b09b-48d7-a396-2afe6dab2f0d\") " Nov 03 22:58:50.840481 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.840454 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-openshift-ca" pod "76a503b3-b09b-48d7-a396-2afe6dab2f0d" (UID: "76a503b3-b09b-48d7-a396-2afe6dab2f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:50.840870 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.840791 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:50.842215 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.842195 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-xks-ca" pod "76a503b3-b09b-48d7-a396-2afe6dab2f0d" (UID: "76a503b3-b09b-48d7-a396-2afe6dab2f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:50.842287 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.842250 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a503b3-b09b-48d7-a396-2afe6dab2f0d-kube-api-access-xmb22" pod "76a503b3-b09b-48d7-a396-2afe6dab2f0d" (UID: "76a503b3-b09b-48d7-a396-2afe6dab2f0d"). InnerVolumeSpecName "kube-api-access-xmb22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:50.942305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.942275 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a503b3-b09b-48d7-a396-2afe6dab2f0d-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:50.942305 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:50.942304 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmb22\" (UniqueName: \"kubernetes.io/projected/76a503b3-b09b-48d7-a396-2afe6dab2f0d-kube-api-access-xmb22\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:58:51.538065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:51.538031 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" event={"ID":"76a503b3-b09b-48d7-a396-2afe6dab2f0d","Type":"ContainerDied","Data":"3309cd4f6a91c2604517add648191bd9eb5b666e0d4ba73d70980cf672af81a9"} Nov 03 22:58:51.538065 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:51.538054 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb" Nov 03 22:58:51.538246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:58:51.538078 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3309cd4f6a91c2604517add648191bd9eb5b666e0d4ba73d70980cf672af81a9" Nov 03 22:58:54.503290 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:58:54.503251 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:59:14.274198 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:14.274156 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 22:59:19.325318 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:19.325251 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt"] Nov 03 22:59:19.325864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:19.325544 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" containerID="cri-o://5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75" gracePeriod=30 Nov 03 22:59:20.737183 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:20.737150 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48"] Nov 03 22:59:20.783331 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:20.783291 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12450a06-dkw48"] Nov 03 22:59:21.494748 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:21.494716 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e70e407-d8fe-4600-842c-68e150e71414" path="/var/lib/kubelet/pods/4e70e407-d8fe-4600-842c-68e150e71414/volumes" Nov 03 22:59:21.514107 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:21.514078 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:59:21.514252 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:21.514137 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:59:24.502054 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:59:24.502022 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:59:24.598257 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.598180 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:59:24.746343 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746311 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-envoy\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746363 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-token\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-token\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746423 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istiod-ca-cert\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746448 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-certs\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746472 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-kube-api-access-qj8g8\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-kube-api-access-qj8g8\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746581 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-podinfo\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746597 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-data\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746722 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-credential-socket\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746756 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-socket\") pod \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\" (UID: \"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2\") " Nov 03 22:59:24.746967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746792 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-certs" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:24.746967 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.746833 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istiod-ca-cert" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:24.747147 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.747080 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-data" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:24.747255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.747230 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-socket" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:24.747542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.747508 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.747638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.747548 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.747638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.747564 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.747638 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.747580 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.747858 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.747703 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-credential-socket" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:24.748692 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.748661 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-envoy" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:24.748785 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.748758 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-kube-api-access-qj8g8" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "kube-api-access-qj8g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:24.748881 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.748778 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-token" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:24.748978 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.748950 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-podinfo" pod "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" (UID: "794b8bf8-0a32-4181-a5cc-0ffa219ae5d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:24.783517 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.783482 5272 generic.go:356] "Generic (PLEG): container finished" podID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerID="5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75" exitCode=0 Nov 03 22:59:24.783656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.783531 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" Nov 03 22:59:24.783656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.783575 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" event={"ID":"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2","Type":"ContainerDied","Data":"5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75"} Nov 03 22:59:24.783656 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.783641 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt" event={"ID":"794b8bf8-0a32-4181-a5cc-0ffa219ae5d2","Type":"ContainerDied","Data":"e9e28e8b39f95686dc6fa0500d29a83d226c341bf8686ccce6629b2bd7410b6c"} Nov 03 22:59:24.783777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.783675 5272 scope.go:122] "RemoveContainer" containerID="5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75" Nov 03 22:59:24.796434 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.796411 5272 scope.go:122] "RemoveContainer" containerID="5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75" Nov 03 22:59:24.796755 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:59:24.796730 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75\": container with ID starting with 5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75 not found: ID does not exist" containerID="5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75" Nov 03 22:59:24.796902 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.796766 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75"} err="failed to get container status \"5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75\": rpc error: code = NotFound desc = could not find container \"5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75\": container with ID starting with 5107025b90ef704cc05a094390254f840a2413202b1f942dcf23374917e91a75 not found: ID does not exist" Nov 03 22:59:24.848548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.848487 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.848548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.848512 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.848548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.848521 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.848548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.848532 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.848548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.848541 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8g8\" (UniqueName: \"kubernetes.io/projected/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2-kube-api-access-qj8g8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:24.977826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:24.977729 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt"] Nov 03 22:59:25.002833 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:25.001040 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-r7xzt"] Nov 03 22:59:25.494954 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:25.494912 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" path="/var/lib/kubelet/pods/794b8bf8-0a32-4181-a5cc-0ffa219ae5d2/volumes" Nov 03 22:59:26.120953 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:26.120921 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv"] Nov 03 22:59:37.793615 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.793576 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf"] Nov 03 22:59:37.794408 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.794386 5272 cpu_manager.go:414] "containerMap: removing container" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" Nov 03 22:59:37.794459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.794410 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" Nov 03 22:59:37.794459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.794440 5272 cpu_manager.go:414] "containerMap: removing container" podUID="76a503b3-b09b-48d7-a396-2afe6dab2f0d" containerName="revoke-keys" Nov 03 22:59:37.794459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.794445 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a503b3-b09b-48d7-a396-2afe6dab2f0d" containerName="revoke-keys" Nov 03 22:59:37.794565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.794553 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="45585d1a-823c-4006-bb93-f257c3dcb895" containerName="istio-proxy" Nov 03 22:59:37.794607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.794570 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a503b3-b09b-48d7-a396-2afe6dab2f0d" containerName="revoke-keys" Nov 03 22:59:37.794607 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.794581 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="794b8bf8-0a32-4181-a5cc-0ffa219ae5d2" containerName="istio-proxy" Nov 03 22:59:37.798039 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.798020 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:37.851567 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.851530 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf"] Nov 03 22:59:37.889254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.888637 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc28bb0-1698-4eaa-9238-0c9313875664-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:37.889254 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:37.888701 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdwb\" (UniqueName: \"kubernetes.io/projected/8cc28bb0-1698-4eaa-9238-0c9313875664-kube-api-access-gpdwb\") pod \"payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:38.068837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.064923 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2cea18-dockercfg-ddtwm\"" Nov 03 22:59:38.068837 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.045929 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2cea18\"" Nov 03 22:59:38.074295 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.074256 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc28bb0-1698-4eaa-9238-0c9313875664-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:38.074474 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.074330 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdwb\" (UniqueName: \"kubernetes.io/projected/8cc28bb0-1698-4eaa-9238-0c9313875664-kube-api-access-gpdwb\") pod \"payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:38.076009 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.075982 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc28bb0-1698-4eaa-9238-0c9313875664-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:38.078125 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.078106 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7"] Nov 03 22:59:38.082466 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.082444 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7"] Nov 03 22:59:38.082547 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.082524 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.132127 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.132102 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdwb\" (UniqueName: \"kubernetes.io/projected/8cc28bb0-1698-4eaa-9238-0c9313875664-kube-api-access-gpdwb\") pod \"payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:38.175482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.175449 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgwh\" (UniqueName: \"kubernetes.io/projected/c6f5ee5b-5fae-4767-b899-7edade51d604-kube-api-access-pdgwh\") pod \"payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.175618 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.175506 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5ee5b-5fae-4767-b899-7edade51d604-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.277459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.277423 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgwh\" (UniqueName: \"kubernetes.io/projected/c6f5ee5b-5fae-4767-b899-7edade51d604-kube-api-access-pdgwh\") pod \"payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.277685 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.277486 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5ee5b-5fae-4767-b899-7edade51d604-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.278251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.278224 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5ee5b-5fae-4767-b899-7edade51d604-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.335320 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.335243 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgwh\" (UniqueName: \"kubernetes.io/projected/c6f5ee5b-5fae-4767-b899-7edade51d604-kube-api-access-pdgwh\") pod \"payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.396489 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.396455 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:38.411911 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.411890 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:38.776747 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.776676 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7"] Nov 03 22:59:38.823066 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:59:38.823036 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f5ee5b_5fae_4767_b899_7edade51d604.slice/crio-ef24d4bc339c975f0930e8165be6d952032ac6e5a569b6b36a050686de44c370 WatchSource:0}: Error finding container ef24d4bc339c975f0930e8165be6d952032ac6e5a569b6b36a050686de44c370: Status 404 returned error can't find the container with id ef24d4bc339c975f0930e8165be6d952032ac6e5a569b6b36a050686de44c370 Nov 03 22:59:38.829145 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:38.829119 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf"] Nov 03 22:59:38.836533 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:59:38.836505 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc28bb0_1698_4eaa_9238_0c9313875664.slice/crio-6309a1d0d348f26e8ef2f9815bdddfadcb101647e9406001d04de08e22c3b327 WatchSource:0}: Error finding container 6309a1d0d348f26e8ef2f9815bdddfadcb101647e9406001d04de08e22c3b327: Status 404 returned error can't find the container with id 6309a1d0d348f26e8ef2f9815bdddfadcb101647e9406001d04de08e22c3b327 Nov 03 22:59:39.083007 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.082967 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" event={"ID":"8cc28bb0-1698-4eaa-9238-0c9313875664","Type":"ContainerStarted","Data":"47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3"} Nov 03 22:59:39.083589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.083032 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" event={"ID":"8cc28bb0-1698-4eaa-9238-0c9313875664","Type":"ContainerStarted","Data":"6309a1d0d348f26e8ef2f9815bdddfadcb101647e9406001d04de08e22c3b327"} Nov 03 22:59:39.083589 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.083112 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:39.084857 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.084797 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" event={"ID":"c6f5ee5b-5fae-4767-b899-7edade51d604","Type":"ContainerStarted","Data":"fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a"} Nov 03 22:59:39.084973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.084860 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" event={"ID":"c6f5ee5b-5fae-4767-b899-7edade51d604","Type":"ContainerStarted","Data":"ef24d4bc339c975f0930e8165be6d952032ac6e5a569b6b36a050686de44c370"} Nov 03 22:59:39.084973 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.084884 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:39.196233 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.196174 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" podStartSLOduration=2.196155359 podStartE2EDuration="2.196155359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:39.191235958 +0000 UTC m=+2159.813544267" watchObservedRunningTime="2026-11-03 22:59:39.196155359 +0000 UTC m=+2159.818463672" Nov 03 22:59:39.302582 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:39.302529 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" podStartSLOduration=2.302513942 podStartE2EDuration="2.302513942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:39.299054384 +0000 UTC m=+2159.921362686" watchObservedRunningTime="2026-11-03 22:59:39.302513942 +0000 UTC m=+2159.924822249" Nov 03 22:59:41.143085 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.143045 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" podUID="f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" containerName="main" containerID="cri-o://40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db" gracePeriod=45 Nov 03 22:59:41.548963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.548937 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:59:41.637189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.635477 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-dshm\") pod \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " Nov 03 22:59:41.637189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.635552 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tmp-dir\") pod \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " Nov 03 22:59:41.637189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.635592 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-home\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-home\") pod \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " Nov 03 22:59:41.637189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.635650 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-model-cache\") pod \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " Nov 03 22:59:41.637189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.635724 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tls-certs\") pod \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " Nov 03 22:59:41.637189 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.635883 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-kube-api-access-vqr5p\" (UniqueName: \"kubernetes.io/projected/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-kube-api-access-vqr5p\") pod \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\" (UID: \"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2\") " Nov 03 22:59:41.640348 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.640048 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-kube-api-access-vqr5p" pod "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" (UID: "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2"). InnerVolumeSpecName "kube-api-access-vqr5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:41.642086 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.641921 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tmp-dir" pod "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" (UID: "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:41.642300 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.642274 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-home" pod "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" (UID: "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:41.642649 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.642626 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-model-cache" pod "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" (UID: "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:41.645132 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.643068 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-dshm" pod "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" (UID: "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:41.647156 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.647128 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tls-certs" pod "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" (UID: "f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:41.738034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.738004 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:41.738034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.738029 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:41.738034 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.738038 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:41.738263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.738047 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:41.738263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.738057 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:41.738263 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:41.738066 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqr5p\" (UniqueName: \"kubernetes.io/projected/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2-kube-api-access-vqr5p\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:42.106988 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.106956 5272 generic.go:356] "Generic (PLEG): container finished" podID="f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" containerID="40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db" exitCode=0 Nov 03 22:59:42.107171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.106999 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" event={"ID":"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2","Type":"ContainerDied","Data":"40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db"} Nov 03 22:59:42.107171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.107011 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" Nov 03 22:59:42.107171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.107032 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv" event={"ID":"f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2","Type":"ContainerDied","Data":"747bda5ab502edfdf81a6a2a16e42a68fd996b4a05b72074ba883c9876043af1"} Nov 03 22:59:42.107171 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.107049 5272 scope.go:122] "RemoveContainer" containerID="40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db" Nov 03 22:59:42.118073 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.118046 5272 scope.go:122] "RemoveContainer" containerID="40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db" Nov 03 22:59:42.118365 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:59:42.118346 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db\": container with ID starting with 40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db not found: ID does not exist" containerID="40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db" Nov 03 22:59:42.118432 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.118375 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db"} err="failed to get container status \"40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db\": rpc error: code = NotFound desc = could not find container \"40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db\": container with ID starting with 40f131366e30bf07984a09b6f8212707a9252059596e9cb211abc9596db158db not found: ID does not exist" Nov 03 22:59:42.278166 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.278131 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv"] Nov 03 22:59:42.322565 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.322534 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-12450a06/e2e-lifecycle-model-12450a06-kserve-66b6f895d5-d6qdv"] Nov 03 22:59:42.980526 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:42.980496 5272 scope.go:122] "RemoveContainer" containerID="61e3d444a9d9a2083a88d752f991c275322348f9b2d2ff00c8a3d3d599daa9bb" Nov 03 22:59:43.494246 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.494212 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" path="/var/lib/kubelet/pods/f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2/volumes" Nov 03 22:59:43.742608 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.742576 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf"] Nov 03 22:59:43.743240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.743211 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" containerName="main" Nov 03 22:59:43.743240 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.743232 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" containerName="main" Nov 03 22:59:43.743460 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.743447 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0aa3a5e-e0f7-44b8-89f4-2556429e0dd2" containerName="main" Nov 03 22:59:43.748637 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.748579 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865389 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865351 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865602 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865394 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865602 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865520 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865602 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865568 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncbc\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-kube-api-access-7ncbc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865645 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865757 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865695 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865779 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865897 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865884 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.865997 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.865981 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.948445 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.948402 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf"] Nov 03 22:59:43.966761 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.966730 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.966963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.966770 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.966963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.966916 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.966963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.966950 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncbc\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-kube-api-access-7ncbc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.966986 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967040 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967136 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967080 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967293 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967163 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967293 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967240 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967554 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967511 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967693 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967595 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967777 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967713 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.967910 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.967886 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.968230 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.968206 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.969814 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.969778 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:43.969951 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:43.969934 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:44.024255 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:44.024190 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:44.024411 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:44.024389 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncbc\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-kube-api-access-7ncbc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ndnzf\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:44.065458 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:44.065430 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:44.497836 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:44.497786 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf"] Nov 03 22:59:44.503780 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:59:44.503746 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf16948_4bf8_472b_b64d_e091ef5f47a4.slice/crio-30b63954e938e524be87b5d03ec08f5e6ed582f0f9b70421fd5ae0d5ece485f5 WatchSource:0}: Error finding container 30b63954e938e524be87b5d03ec08f5e6ed582f0f9b70421fd5ae0d5ece485f5: Status 404 returned error can't find the container with id 30b63954e938e524be87b5d03ec08f5e6ed582f0f9b70421fd5ae0d5ece485f5 Nov 03 22:59:44.506491 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:44.506450 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:44.506600 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:44.506539 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:44.506666 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:44.506607 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:59:45.134683 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:45.134644 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" event={"ID":"9cf16948-4bf8-472b-b64d-e091ef5f47a4","Type":"ContainerStarted","Data":"72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36"} Nov 03 22:59:45.134901 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:45.134705 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" event={"ID":"9cf16948-4bf8-472b-b64d-e091ef5f47a4","Type":"ContainerStarted","Data":"30b63954e938e524be87b5d03ec08f5e6ed582f0f9b70421fd5ae0d5ece485f5"} Nov 03 22:59:45.342459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:45.342400 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podStartSLOduration=2.342386353 podStartE2EDuration="2.342386353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:45.294000704 +0000 UTC m=+2165.916309012" watchObservedRunningTime="2026-11-03 22:59:45.342386353 +0000 UTC m=+2165.964694661" Nov 03 22:59:46.068032 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:46.067990 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:46.069887 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:46.069858 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ndnzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:59:46.070019 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:46.069926 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:59:47.071826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:47.066413 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ndnzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:59:47.071826 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:47.066476 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:59:48.071832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:48.070520 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ndnzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:59:48.071832 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:48.070580 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:59:49.068223 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:49.068182 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ndnzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:59:49.068442 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:49.068249 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:59:49.236864 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:49.236787 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln"] Nov 03 22:59:49.284292 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:49.284251 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2627c5-rxzln"] Nov 03 22:59:49.496072 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:49.495994 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdc955a-fa53-4214-90d2-c14f34606a61" path="/var/lib/kubelet/pods/ebdc955a-fa53-4214-90d2-c14f34606a61/volumes" Nov 03 22:59:50.068276 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:50.068183 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ndnzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:59:50.068717 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:50.068676 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:59:50.094235 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:50.094210 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 22:59:50.094339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:50.094290 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 22:59:51.066061 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:51.066025 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ndnzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:59:51.066459 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:51.066092 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:59:52.066328 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:52.066298 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ndnzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 03 22:59:52.066713 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:52.066349 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 03 22:59:53.116482 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:53.116439 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:53.117202 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:53.117172 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:53.117962 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:53.117936 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 22:59:53.963963 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:53.963919 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4"] Nov 03 22:59:53.968965 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:53.968940 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.023922 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.023884 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4"] Nov 03 22:59:54.025301 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.025266 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:54.025301 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.025282 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:59:54.025501 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.025403 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:59:54.025556 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.025507 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 22:59:54.073548 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.073511 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.073730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.073587 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4d6j\" (UniqueName: \"kubernetes.io/projected/0449308a-4743-4b1e-94bd-fa1cf13cdca7-kube-api-access-r4d6j\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.073772 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.073716 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.175251 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.175215 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.175646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.175298 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4d6j\" (UniqueName: \"kubernetes.io/projected/0449308a-4743-4b1e-94bd-fa1cf13cdca7-kube-api-access-r4d6j\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.175646 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.175363 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.175909 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.175890 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.177730 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.177705 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.235689 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.235612 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4d6j\" (UniqueName: \"kubernetes.io/projected/0449308a-4743-4b1e-94bd-fa1cf13cdca7-kube-api-access-r4d6j\") pod \"maas-api-revoke-keys-e2e-api-b-d6296b-kflt4\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.294712 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.294679 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:54.505738 ip-10-0-128-222 kubenswrapper[5272]: E1103 22:59:54.505650 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 22:59:54.676542 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:54.676496 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4"] Nov 03 22:59:54.682657 ip-10-0-128-222 kubenswrapper[5272]: W1103 22:59:54.682624 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0449308a_4743_4b1e_94bd_fa1cf13cdca7.slice/crio-ab2a9e428d8fc1417ec433023cb448886e9a7d4ee04f9c642e736f41c2b37a83 WatchSource:0}: Error finding container ab2a9e428d8fc1417ec433023cb448886e9a7d4ee04f9c642e736f41c2b37a83: Status 404 returned error can't find the container with id ab2a9e428d8fc1417ec433023cb448886e9a7d4ee04f9c642e736f41c2b37a83 Nov 03 22:59:55.276177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:55.276090 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" event={"ID":"0449308a-4743-4b1e-94bd-fa1cf13cdca7","Type":"ContainerStarted","Data":"65dd545d32bbaf27c8783313a23dd267d6499f304e6b0f0a8dcfbd07a4509bae"} Nov 03 22:59:55.276177 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:55.276148 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" event={"ID":"0449308a-4743-4b1e-94bd-fa1cf13cdca7","Type":"ContainerStarted","Data":"ab2a9e428d8fc1417ec433023cb448886e9a7d4ee04f9c642e736f41c2b37a83"} Nov 03 22:59:55.385393 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:55.385342 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" podStartSLOduration=2.385327852 podStartE2EDuration="2.385327852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:55.383947078 +0000 UTC m=+2176.006255386" watchObservedRunningTime="2026-11-03 22:59:55.385327852 +0000 UTC m=+2176.007636158" Nov 03 22:59:56.284339 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:56.284304 5272 generic.go:356] "Generic (PLEG): container finished" podID="0449308a-4743-4b1e-94bd-fa1cf13cdca7" containerID="65dd545d32bbaf27c8783313a23dd267d6499f304e6b0f0a8dcfbd07a4509bae" exitCode=0 Nov 03 22:59:56.284863 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:56.284349 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" event={"ID":"0449308a-4743-4b1e-94bd-fa1cf13cdca7","Type":"ContainerDied","Data":"65dd545d32bbaf27c8783313a23dd267d6499f304e6b0f0a8dcfbd07a4509bae"} Nov 03 22:59:57.453497 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.453468 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:57.524651 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.524622 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-openshift-ca\") pod \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " Nov 03 22:59:57.524838 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.524789 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-xks-ca\") pod \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " Nov 03 22:59:57.524895 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.524884 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0449308a-4743-4b1e-94bd-fa1cf13cdca7-kube-api-access-r4d6j\" (UniqueName: \"kubernetes.io/projected/0449308a-4743-4b1e-94bd-fa1cf13cdca7-kube-api-access-r4d6j\") pod \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\" (UID: \"0449308a-4743-4b1e-94bd-fa1cf13cdca7\") " Nov 03 22:59:57.525031 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.525001 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-openshift-ca" pod "0449308a-4743-4b1e-94bd-fa1cf13cdca7" (UID: "0449308a-4743-4b1e-94bd-fa1cf13cdca7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:57.525330 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.525307 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:57.526860 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.526833 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-xks-ca" pod "0449308a-4743-4b1e-94bd-fa1cf13cdca7" (UID: "0449308a-4743-4b1e-94bd-fa1cf13cdca7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:57.527108 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.527087 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0449308a-4743-4b1e-94bd-fa1cf13cdca7-kube-api-access-r4d6j" pod "0449308a-4743-4b1e-94bd-fa1cf13cdca7" (UID: "0449308a-4743-4b1e-94bd-fa1cf13cdca7"). InnerVolumeSpecName "kube-api-access-r4d6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:57.634883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.627903 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0449308a-4743-4b1e-94bd-fa1cf13cdca7-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:57.634883 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:57.627946 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4d6j\" (UniqueName: \"kubernetes.io/projected/0449308a-4743-4b1e-94bd-fa1cf13cdca7-kube-api-access-r4d6j\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:59:58.300316 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:58.300277 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" Nov 03 22:59:58.300500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:58.300324 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4" event={"ID":"0449308a-4743-4b1e-94bd-fa1cf13cdca7","Type":"ContainerDied","Data":"ab2a9e428d8fc1417ec433023cb448886e9a7d4ee04f9c642e736f41c2b37a83"} Nov 03 22:59:58.300500 ip-10-0-128-222 kubenswrapper[5272]: I1103 22:59:58.300373 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab2a9e428d8fc1417ec433023cb448886e9a7d4ee04f9c642e736f41c2b37a83" Nov 03 23:00:00.169250 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.169214 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4"] Nov 03 23:00:00.170393 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.170367 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0449308a-4743-4b1e-94bd-fa1cf13cdca7" containerName="revoke-keys" Nov 03 23:00:00.170393 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.170393 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0449308a-4743-4b1e-94bd-fa1cf13cdca7" containerName="revoke-keys" Nov 03 23:00:00.170581 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.170555 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="0449308a-4743-4b1e-94bd-fa1cf13cdca7" containerName="revoke-keys" Nov 03 23:00:00.174587 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.174566 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp"] Nov 03 23:00:00.174706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.174689 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.184330 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.184299 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8"] Nov 03 23:00:00.187187 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.187161 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.191158 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.191136 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.216169 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.216127 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw"] Nov 03 23:00:00.220606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.220585 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk"] Nov 03 23:00:00.220714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.220696 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.224924 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.224905 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94"] Nov 03 23:00:00.225039 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.225021 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.229199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.229179 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc"] Nov 03 23:00:00.229284 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.229271 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.234250 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.234230 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw"] Nov 03 23:00:00.234350 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.234331 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.238644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.238623 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.259257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.259214 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:00:00.259381 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.259267 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:00:00.259381 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.259367 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:00:00.262856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.262835 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn"] Nov 03 23:00:00.267967 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.267950 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb"] Nov 03 23:00:00.268086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.268070 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.270860 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.270833 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126243d7-7eaa-47db-b2da-615c6559e529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.270982 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.270906 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38737a60-c402-4a2a-b52a-d765dcc36e1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.271065 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.271045 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.271127 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.271084 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzf48\" (UniqueName: \"kubernetes.io/projected/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-kube-api-access-fzf48\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.271197 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.271184 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrtb\" (UniqueName: \"kubernetes.io/projected/38737a60-c402-4a2a-b52a-d765dcc36e1c-kube-api-access-dlrtb\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.271273 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.271257 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9lk\" (UniqueName: \"kubernetes.io/projected/126243d7-7eaa-47db-b2da-615c6559e529-kube-api-access-9v9lk\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.274220 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.274074 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4"] Nov 03 23:00:00.274220 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.274198 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.275663 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.275642 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw"] Nov 03 23:00:00.278684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.278664 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp"] Nov 03 23:00:00.322196 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.316396 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk"] Nov 03 23:00:00.322196 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.322071 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc"] Nov 03 23:00:00.326020 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.325991 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw"] Nov 03 23:00:00.358895 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.358864 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn"] Nov 03 23:00:00.368580 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.368545 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8"] Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376082 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d767797-1338-49e6-b3ad-57fa05f249b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-dv2fb\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376133 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrtb\" (UniqueName: \"kubernetes.io/projected/38737a60-c402-4a2a-b52a-d765dcc36e1c-kube-api-access-dlrtb\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376190 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec2ed53-af75-461b-872c-27f9b7bef027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376269 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9lk\" (UniqueName: \"kubernetes.io/projected/126243d7-7eaa-47db-b2da-615c6559e529-kube-api-access-9v9lk\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376350 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svx2d\" (UniqueName: \"kubernetes.io/projected/641c211c-d139-4894-ae2f-26e70a843973-kube-api-access-svx2d\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376385 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed28ee9e-6798-4f37-8020-5325aa4c63cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376477 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8nn\" (UniqueName: \"kubernetes.io/projected/fc425062-2635-4d50-81fc-56bd9a9dab42-kube-api-access-9b8nn\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376503 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fj4\" (UniqueName: \"kubernetes.io/projected/8d6781d3-1e50-4b96-8874-2dc962a89a4f-kube-api-access-d5fj4\") pod \"maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376533 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126243d7-7eaa-47db-b2da-615c6559e529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376581 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38737a60-c402-4a2a-b52a-d765dcc36e1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376628 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfmw\" (UniqueName: \"kubernetes.io/projected/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-kube-api-access-9cfmw\") pod \"maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376648 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94"] Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376679 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zjp\" (UniqueName: \"kubernetes.io/projected/ed28ee9e-6798-4f37-8020-5325aa4c63cb-kube-api-access-77zjp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376713 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376743 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6781d3-1e50-4b96-8874-2dc962a89a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376786 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfsf6\" (UniqueName: \"kubernetes.io/projected/eec2ed53-af75-461b-872c-27f9b7bef027-kube-api-access-nfsf6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376832 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376860 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzf48\" (UniqueName: \"kubernetes.io/projected/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-kube-api-access-fzf48\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376885 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tn8\" (UniqueName: \"kubernetes.io/projected/5d767797-1338-49e6-b3ad-57fa05f249b5-kube-api-access-48tn8\") pod \"maas-api-key-cleanup-29895780-dv2fb\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376934 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc425062-2635-4d50-81fc-56bd9a9dab42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.376961 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c211c-d139-4894-ae2f-26e70a843973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.378315 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126243d7-7eaa-47db-b2da-615c6559e529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.379004 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38737a60-c402-4a2a-b52a-d765dcc36e1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.379797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.379588 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.382541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.382518 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb"] Nov 03 23:00:00.435354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.435285 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9lk\" (UniqueName: \"kubernetes.io/projected/126243d7-7eaa-47db-b2da-615c6559e529-kube-api-access-9v9lk\") pod \"maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.435354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.435303 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrtb\" (UniqueName: \"kubernetes.io/projected/38737a60-c402-4a2a-b52a-d765dcc36e1c-kube-api-access-dlrtb\") pod \"maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.436259 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.436237 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzf48\" (UniqueName: \"kubernetes.io/projected/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-kube-api-access-fzf48\") pod \"maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.478119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478092 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx2d\" (UniqueName: \"kubernetes.io/projected/641c211c-d139-4894-ae2f-26e70a843973-kube-api-access-svx2d\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.478276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478130 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed28ee9e-6798-4f37-8020-5325aa4c63cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.478276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478222 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8nn\" (UniqueName: \"kubernetes.io/projected/fc425062-2635-4d50-81fc-56bd9a9dab42-kube-api-access-9b8nn\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.478276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478262 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fj4\" (UniqueName: \"kubernetes.io/projected/8d6781d3-1e50-4b96-8874-2dc962a89a4f-kube-api-access-d5fj4\") pod \"maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.478464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478333 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfmw\" (UniqueName: \"kubernetes.io/projected/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-kube-api-access-9cfmw\") pod \"maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.478464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478385 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zjp\" (UniqueName: \"kubernetes.io/projected/ed28ee9e-6798-4f37-8020-5325aa4c63cb-kube-api-access-77zjp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.478464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478417 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.478464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478451 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6781d3-1e50-4b96-8874-2dc962a89a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.478685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478505 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfsf6\" (UniqueName: \"kubernetes.io/projected/eec2ed53-af75-461b-872c-27f9b7bef027-kube-api-access-nfsf6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.478685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478537 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tn8\" (UniqueName: \"kubernetes.io/projected/5d767797-1338-49e6-b3ad-57fa05f249b5-kube-api-access-48tn8\") pod \"maas-api-key-cleanup-29895780-dv2fb\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.478685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478667 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc425062-2635-4d50-81fc-56bd9a9dab42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.478885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478703 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c211c-d139-4894-ae2f-26e70a843973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.478885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478766 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d767797-1338-49e6-b3ad-57fa05f249b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-dv2fb\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.479013 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478867 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec2ed53-af75-461b-872c-27f9b7bef027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.479013 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.478900 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed28ee9e-6798-4f37-8020-5325aa4c63cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.479245 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.479221 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6781d3-1e50-4b96-8874-2dc962a89a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.479328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.479256 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.479506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.479481 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc425062-2635-4d50-81fc-56bd9a9dab42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.479583 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.479509 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c211c-d139-4894-ae2f-26e70a843973-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.479667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.479582 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d767797-1338-49e6-b3ad-57fa05f249b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-dv2fb\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.479667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.479619 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec2ed53-af75-461b-872c-27f9b7bef027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.496407 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.496385 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:00.507941 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.507923 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:00.514093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.514076 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:00.537046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.537006 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fj4\" (UniqueName: \"kubernetes.io/projected/8d6781d3-1e50-4b96-8874-2dc962a89a4f-kube-api-access-d5fj4\") pod \"maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.537627 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.537595 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zjp\" (UniqueName: \"kubernetes.io/projected/ed28ee9e-6798-4f37-8020-5325aa4c63cb-kube-api-access-77zjp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.541933 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.541900 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfsf6\" (UniqueName: \"kubernetes.io/projected/eec2ed53-af75-461b-872c-27f9b7bef027-kube-api-access-nfsf6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.543213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.543190 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfmw\" (UniqueName: \"kubernetes.io/projected/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-kube-api-access-9cfmw\") pod \"maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.543556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.543526 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tn8\" (UniqueName: \"kubernetes.io/projected/5d767797-1338-49e6-b3ad-57fa05f249b5-kube-api-access-48tn8\") pod \"maas-api-key-cleanup-29895780-dv2fb\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.545586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.545552 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx2d\" (UniqueName: \"kubernetes.io/projected/641c211c-d139-4894-ae2f-26e70a843973-kube-api-access-svx2d\") pod \"maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.545972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.545939 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8nn\" (UniqueName: \"kubernetes.io/projected/fc425062-2635-4d50-81fc-56bd9a9dab42-kube-api-access-9b8nn\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.552469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.552448 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:00.557298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.557278 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:00.578714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.577939 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:00.578714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.578660 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:00.591493 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.591466 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:00.593522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.593155 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:00.837093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.837063 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:00.899930 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:00.899774 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4"] Nov 03 23:00:01.112059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.111939 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp"] Nov 03 23:00:01.117563 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:00:01.117535 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38737a60_c402_4a2a_b52a_d765dcc36e1c.slice/crio-bff65f6665a8e8105f05440bfebfa84fb6d8221273a04b7d23daacf688387d49 WatchSource:0}: Error finding container bff65f6665a8e8105f05440bfebfa84fb6d8221273a04b7d23daacf688387d49: Status 404 returned error can't find the container with id bff65f6665a8e8105f05440bfebfa84fb6d8221273a04b7d23daacf688387d49 Nov 03 23:00:01.230629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.230600 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc"] Nov 03 23:00:01.234606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.234576 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw"] Nov 03 23:00:01.236506 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:00:01.236478 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641c211c_d139_4894_ae2f_26e70a843973.slice/crio-50931d11b2e3510c939e3dd9221d984c711582f796b47d9a759d872e8c8d5b80 WatchSource:0}: Error finding container 50931d11b2e3510c939e3dd9221d984c711582f796b47d9a759d872e8c8d5b80: Status 404 returned error can't find the container with id 50931d11b2e3510c939e3dd9221d984c711582f796b47d9a759d872e8c8d5b80 Nov 03 23:00:01.239062 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.239041 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk"] Nov 03 23:00:01.239148 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:00:01.239054 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded28ee9e_6798_4f37_8020_5325aa4c63cb.slice/crio-47b23ee9310f5acbaeadb1acb1345ada5b9c2b05524779876976956089ea549d WatchSource:0}: Error finding container 47b23ee9310f5acbaeadb1acb1345ada5b9c2b05524779876976956089ea549d: Status 404 returned error can't find the container with id 47b23ee9310f5acbaeadb1acb1345ada5b9c2b05524779876976956089ea549d Nov 03 23:00:01.244965 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:00:01.244939 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec2ed53_af75_461b_872c_27f9b7bef027.slice/crio-a81b195dfab0387a89a9286876a8c8efd9d788f12ed117339442e521cdc6f247 WatchSource:0}: Error finding container a81b195dfab0387a89a9286876a8c8efd9d788f12ed117339442e521cdc6f247: Status 404 returned error can't find the container with id a81b195dfab0387a89a9286876a8c8efd9d788f12ed117339442e521cdc6f247 Nov 03 23:00:01.275497 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.275467 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb"] Nov 03 23:00:01.281200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.281178 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94"] Nov 03 23:00:01.288246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.288224 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8"] Nov 03 23:00:01.311705 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.311679 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn"] Nov 03 23:00:01.327689 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.327666 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw"] Nov 03 23:00:01.332866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.332842 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" event={"ID":"641c211c-d139-4894-ae2f-26e70a843973","Type":"ContainerStarted","Data":"50931d11b2e3510c939e3dd9221d984c711582f796b47d9a759d872e8c8d5b80"} Nov 03 23:00:01.334235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.334208 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" event={"ID":"38737a60-c402-4a2a-b52a-d765dcc36e1c","Type":"ContainerStarted","Data":"bff65f6665a8e8105f05440bfebfa84fb6d8221273a04b7d23daacf688387d49"} Nov 03 23:00:01.335634 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.335610 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" event={"ID":"ed28ee9e-6798-4f37-8020-5325aa4c63cb","Type":"ContainerStarted","Data":"47b23ee9310f5acbaeadb1acb1345ada5b9c2b05524779876976956089ea549d"} Nov 03 23:00:01.336830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.336797 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" event={"ID":"7d909e24-6172-4e2a-b04e-8a0c7c203a0f","Type":"ContainerStarted","Data":"edd321200a9b089c5ae9a453d3906c97a60ffee9d89bd5eabf5f714777f4ef23"} Nov 03 23:00:01.337958 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:01.337938 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerStarted","Data":"a81b195dfab0387a89a9286876a8c8efd9d788f12ed117339442e521cdc6f247"} Nov 03 23:00:01.349716 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:00:01.346028 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d767797_1338_49e6_b3ad_57fa05f249b5.slice/crio-f2897206e47a1643aedecb12aaa6f998e1749cb8ec2ee98ace543150decb92da WatchSource:0}: Error finding container f2897206e47a1643aedecb12aaa6f998e1749cb8ec2ee98ace543150decb92da: Status 404 returned error can't find the container with id f2897206e47a1643aedecb12aaa6f998e1749cb8ec2ee98ace543150decb92da Nov 03 23:00:01.360986 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:00:01.360946 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc425062_2635_4d50_81fc_56bd9a9dab42.slice/crio-d04cc1f326e732e42aee9a41d6eff9dd3bd487cb347019bf9211c3c7ee105cbe WatchSource:0}: Error finding container d04cc1f326e732e42aee9a41d6eff9dd3bd487cb347019bf9211c3c7ee105cbe: Status 404 returned error can't find the container with id d04cc1f326e732e42aee9a41d6eff9dd3bd487cb347019bf9211c3c7ee105cbe Nov 03 23:00:01.361240 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:00:01.361207 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6781d3_1e50_4b96_8874_2dc962a89a4f.slice/crio-4ae81754972267163bfd5a0f07841e315e36a986a100bf7f2beca2b5e04e689a WatchSource:0}: Error finding container 4ae81754972267163bfd5a0f07841e315e36a986a100bf7f2beca2b5e04e689a: Status 404 returned error can't find the container with id 4ae81754972267163bfd5a0f07841e315e36a986a100bf7f2beca2b5e04e689a Nov 03 23:00:02.351755 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.351686 5272 generic.go:356] "Generic (PLEG): container finished" podID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerID="df8c9de8d3b711ec092e9c431019dbf910410a42512a371379c27084a0f1afef" exitCode=60 Nov 03 23:00:02.352765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.352104 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" event={"ID":"7d909e24-6172-4e2a-b04e-8a0c7c203a0f","Type":"ContainerDied","Data":"df8c9de8d3b711ec092e9c431019dbf910410a42512a371379c27084a0f1afef"} Nov 03 23:00:02.353916 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.353892 5272 scope.go:122] "RemoveContainer" containerID="df8c9de8d3b711ec092e9c431019dbf910410a42512a371379c27084a0f1afef" Nov 03 23:00:02.356588 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.356563 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerStarted","Data":"7665648b1e8362d57df7029d3ef32a345ad479d70455729b2692b5b75c1c7709"} Nov 03 23:00:02.358424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.358403 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" event={"ID":"641c211c-d139-4894-ae2f-26e70a843973","Type":"ContainerStarted","Data":"d171fc5f314b6c85a1cf0735a949b9bead4bf7b325cdec63a256f890cf2ed7a7"} Nov 03 23:00:02.364873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.364845 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" event={"ID":"fc425062-2635-4d50-81fc-56bd9a9dab42","Type":"ContainerStarted","Data":"6ebf8e288d03bbd738850203215c38e88d550cbbbe1f3cd911a70bf7ade65a6f"} Nov 03 23:00:02.364990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.364893 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" event={"ID":"fc425062-2635-4d50-81fc-56bd9a9dab42","Type":"ContainerStarted","Data":"d04cc1f326e732e42aee9a41d6eff9dd3bd487cb347019bf9211c3c7ee105cbe"} Nov 03 23:00:02.366486 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.366462 5272 generic.go:356] "Generic (PLEG): container finished" podID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerID="80985a8fedc16b96fc2d239dc51b9d556e6393966e5e39206652918170ae280a" exitCode=60 Nov 03 23:00:02.366597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.366530 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" event={"ID":"38737a60-c402-4a2a-b52a-d765dcc36e1c","Type":"ContainerDied","Data":"80985a8fedc16b96fc2d239dc51b9d556e6393966e5e39206652918170ae280a"} Nov 03 23:00:02.366735 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.366721 5272 scope.go:122] "RemoveContainer" containerID="80985a8fedc16b96fc2d239dc51b9d556e6393966e5e39206652918170ae280a" Nov 03 23:00:02.368130 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.368106 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" event={"ID":"5d767797-1338-49e6-b3ad-57fa05f249b5","Type":"ContainerStarted","Data":"dfcb2d2966da08f5f51b395aa70b855e1a817db2380fe4b179151a4295c1881e"} Nov 03 23:00:02.368241 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.368144 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" event={"ID":"5d767797-1338-49e6-b3ad-57fa05f249b5","Type":"ContainerStarted","Data":"f2897206e47a1643aedecb12aaa6f998e1749cb8ec2ee98ace543150decb92da"} Nov 03 23:00:02.370516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.370487 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" event={"ID":"09966415-3039-4b0f-8ab5-47f1d1e8cdf0","Type":"ContainerStarted","Data":"32a47e38188f51e0f401e45f90cc13bf31a41206ec2fd1e4c843a00ba02a4f12"} Nov 03 23:00:02.370592 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.370536 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" event={"ID":"09966415-3039-4b0f-8ab5-47f1d1e8cdf0","Type":"ContainerStarted","Data":"02563bdf3354f4e23315939d87e7ef8e6c5714ade59a24dd71eb864a8b1fd49f"} Nov 03 23:00:02.373415 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.373388 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" event={"ID":"126243d7-7eaa-47db-b2da-615c6559e529","Type":"ContainerStarted","Data":"dcb35cddbc4853e70acc38fcc8ac04ce859077c56c2a2693113ab835ef0c7f95"} Nov 03 23:00:02.373522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.373430 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" event={"ID":"126243d7-7eaa-47db-b2da-615c6559e529","Type":"ContainerStarted","Data":"1b6b7f1fb71730c75fb9636f2a26a45a1a7a10f4eccf71d3a27672b4a3e1b0f2"} Nov 03 23:00:02.375079 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.375045 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" event={"ID":"ed28ee9e-6798-4f37-8020-5325aa4c63cb","Type":"ContainerStarted","Data":"58f2728a87e592b384a0424c7e8a0e7e2296934331b33122448d31cf98cac618"} Nov 03 23:00:02.376604 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.376578 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" event={"ID":"8d6781d3-1e50-4b96-8874-2dc962a89a4f","Type":"ContainerStarted","Data":"e6e719da2ba10b92f9542468cdb580f46dc43fb8de64bc6c46e3a942ada90496"} Nov 03 23:00:02.376702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.376624 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" event={"ID":"8d6781d3-1e50-4b96-8874-2dc962a89a4f","Type":"ContainerStarted","Data":"4ae81754972267163bfd5a0f07841e315e36a986a100bf7f2beca2b5e04e689a"} Nov 03 23:00:02.579329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.579285 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" podStartSLOduration=2.579270319 podStartE2EDuration="2.579270319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.575357893 +0000 UTC m=+2183.197666202" watchObservedRunningTime="2026-11-03 23:00:02.579270319 +0000 UTC m=+2183.201578627" Nov 03 23:00:02.799798 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.799751 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" podStartSLOduration=2.799738028 podStartE2EDuration="2.799738028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.687929004 +0000 UTC m=+2183.310237316" watchObservedRunningTime="2026-11-03 23:00:02.799738028 +0000 UTC m=+2183.422046387" Nov 03 23:00:02.800016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.799996 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" podStartSLOduration=2.799990345 podStartE2EDuration="2.799990345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.797031805 +0000 UTC m=+2183.419340115" watchObservedRunningTime="2026-11-03 23:00:02.799990345 +0000 UTC m=+2183.422298652" Nov 03 23:00:02.955510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:02.955454 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" podStartSLOduration=2.955438415 podStartE2EDuration="2.955438415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.953179982 +0000 UTC m=+2183.575488289" watchObservedRunningTime="2026-11-03 23:00:02.955438415 +0000 UTC m=+2183.577746724" Nov 03 23:00:03.064413 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.064369 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" podStartSLOduration=3.064357129 podStartE2EDuration="3.064357129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.062658795 +0000 UTC m=+2183.684967102" watchObservedRunningTime="2026-11-03 23:00:03.064357129 +0000 UTC m=+2183.686665431" Nov 03 23:00:03.280830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.280706 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" podStartSLOduration=3.280691529 podStartE2EDuration="3.280691529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.279191489 +0000 UTC m=+2183.901499797" watchObservedRunningTime="2026-11-03 23:00:03.280691529 +0000 UTC m=+2183.902999836" Nov 03 23:00:03.387197 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.387159 5272 generic.go:356] "Generic (PLEG): container finished" podID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerID="58f2728a87e592b384a0424c7e8a0e7e2296934331b33122448d31cf98cac618" exitCode=60 Nov 03 23:00:03.387649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.387235 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" event={"ID":"ed28ee9e-6798-4f37-8020-5325aa4c63cb","Type":"ContainerDied","Data":"58f2728a87e592b384a0424c7e8a0e7e2296934331b33122448d31cf98cac618"} Nov 03 23:00:03.387649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.387522 5272 scope.go:122] "RemoveContainer" containerID="58f2728a87e592b384a0424c7e8a0e7e2296934331b33122448d31cf98cac618" Nov 03 23:00:03.389430 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.389170 5272 generic.go:356] "Generic (PLEG): container finished" podID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerID="e6e719da2ba10b92f9542468cdb580f46dc43fb8de64bc6c46e3a942ada90496" exitCode=60 Nov 03 23:00:03.389430 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.389245 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" event={"ID":"8d6781d3-1e50-4b96-8874-2dc962a89a4f","Type":"ContainerDied","Data":"e6e719da2ba10b92f9542468cdb580f46dc43fb8de64bc6c46e3a942ada90496"} Nov 03 23:00:03.389553 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.389520 5272 scope.go:122] "RemoveContainer" containerID="e6e719da2ba10b92f9542468cdb580f46dc43fb8de64bc6c46e3a942ada90496" Nov 03 23:00:03.391473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.391393 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" podStartSLOduration=3.391380921 podStartE2EDuration="3.391380921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.389054283 +0000 UTC m=+2184.011362590" watchObservedRunningTime="2026-11-03 23:00:03.391380921 +0000 UTC m=+2184.013689228" Nov 03 23:00:03.392552 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.392511 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" event={"ID":"7d909e24-6172-4e2a-b04e-8a0c7c203a0f","Type":"ContainerStarted","Data":"c1f3a919ac94aaf98d3075d3deefdce7f67da99e837316de2f19363c9332908f"} Nov 03 23:00:03.394484 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.394463 5272 generic.go:356] "Generic (PLEG): container finished" podID="eec2ed53-af75-461b-872c-27f9b7bef027" containerID="7665648b1e8362d57df7029d3ef32a345ad479d70455729b2692b5b75c1c7709" exitCode=60 Nov 03 23:00:03.394586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.394544 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerDied","Data":"7665648b1e8362d57df7029d3ef32a345ad479d70455729b2692b5b75c1c7709"} Nov 03 23:00:03.394736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.394721 5272 scope.go:122] "RemoveContainer" containerID="7665648b1e8362d57df7029d3ef32a345ad479d70455729b2692b5b75c1c7709" Nov 03 23:00:03.396042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.395997 5272 generic.go:356] "Generic (PLEG): container finished" podID="641c211c-d139-4894-ae2f-26e70a843973" containerID="d171fc5f314b6c85a1cf0735a949b9bead4bf7b325cdec63a256f890cf2ed7a7" exitCode=60 Nov 03 23:00:03.396172 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.396081 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" event={"ID":"641c211c-d139-4894-ae2f-26e70a843973","Type":"ContainerDied","Data":"d171fc5f314b6c85a1cf0735a949b9bead4bf7b325cdec63a256f890cf2ed7a7"} Nov 03 23:00:03.396636 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.396304 5272 scope.go:122] "RemoveContainer" containerID="d171fc5f314b6c85a1cf0735a949b9bead4bf7b325cdec63a256f890cf2ed7a7" Nov 03 23:00:03.397916 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.397896 5272 generic.go:356] "Generic (PLEG): container finished" podID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerID="6ebf8e288d03bbd738850203215c38e88d550cbbbe1f3cd911a70bf7ade65a6f" exitCode=60 Nov 03 23:00:03.397991 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.397973 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" event={"ID":"fc425062-2635-4d50-81fc-56bd9a9dab42","Type":"ContainerDied","Data":"6ebf8e288d03bbd738850203215c38e88d550cbbbe1f3cd911a70bf7ade65a6f"} Nov 03 23:00:03.398189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.398173 5272 scope.go:122] "RemoveContainer" containerID="6ebf8e288d03bbd738850203215c38e88d550cbbbe1f3cd911a70bf7ade65a6f" Nov 03 23:00:03.400326 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.400183 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" event={"ID":"38737a60-c402-4a2a-b52a-d765dcc36e1c","Type":"ContainerStarted","Data":"926d110a37ea6628c576b6d7d3b3776e330cba83305c5caf29057add9915e42a"} Nov 03 23:00:03.402035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.402011 5272 generic.go:356] "Generic (PLEG): container finished" podID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerID="dfcb2d2966da08f5f51b395aa70b855e1a817db2380fe4b179151a4295c1881e" exitCode=60 Nov 03 23:00:03.402137 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.402090 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" event={"ID":"5d767797-1338-49e6-b3ad-57fa05f249b5","Type":"ContainerDied","Data":"dfcb2d2966da08f5f51b395aa70b855e1a817db2380fe4b179151a4295c1881e"} Nov 03 23:00:03.402276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.402261 5272 scope.go:122] "RemoveContainer" containerID="dfcb2d2966da08f5f51b395aa70b855e1a817db2380fe4b179151a4295c1881e" Nov 03 23:00:03.403641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.403625 5272 generic.go:356] "Generic (PLEG): container finished" podID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerID="32a47e38188f51e0f401e45f90cc13bf31a41206ec2fd1e4c843a00ba02a4f12" exitCode=60 Nov 03 23:00:03.403740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.403682 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" event={"ID":"09966415-3039-4b0f-8ab5-47f1d1e8cdf0","Type":"ContainerDied","Data":"32a47e38188f51e0f401e45f90cc13bf31a41206ec2fd1e4c843a00ba02a4f12"} Nov 03 23:00:03.403964 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.403946 5272 scope.go:122] "RemoveContainer" containerID="32a47e38188f51e0f401e45f90cc13bf31a41206ec2fd1e4c843a00ba02a4f12" Nov 03 23:00:03.405500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.405481 5272 generic.go:356] "Generic (PLEG): container finished" podID="126243d7-7eaa-47db-b2da-615c6559e529" containerID="dcb35cddbc4853e70acc38fcc8ac04ce859077c56c2a2693113ab835ef0c7f95" exitCode=60 Nov 03 23:00:03.405607 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.405522 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" event={"ID":"126243d7-7eaa-47db-b2da-615c6559e529","Type":"ContainerDied","Data":"dcb35cddbc4853e70acc38fcc8ac04ce859077c56c2a2693113ab835ef0c7f95"} Nov 03 23:00:03.405711 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.405698 5272 scope.go:122] "RemoveContainer" containerID="dcb35cddbc4853e70acc38fcc8ac04ce859077c56c2a2693113ab835ef0c7f95" Nov 03 23:00:03.499889 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:03.499845 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" podStartSLOduration=3.499830798 podStartE2EDuration="3.499830798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.495371545 +0000 UTC m=+2184.117679850" watchObservedRunningTime="2026-11-03 23:00:03.499830798 +0000 UTC m=+2184.122139133" Nov 03 23:00:04.374067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.373991 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" podStartSLOduration=4.373978414 podStartE2EDuration="4.373978414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:04.37141111 +0000 UTC m=+2184.993719419" watchObservedRunningTime="2026-11-03 23:00:04.373978414 +0000 UTC m=+2184.996286722" Nov 03 23:00:04.415028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.414989 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" event={"ID":"fc425062-2635-4d50-81fc-56bd9a9dab42","Type":"ContainerStarted","Data":"3f32e0eb89b802f85b03f4f4a0c12254df8c64bad0e749fbd979dadc799d98c8"} Nov 03 23:00:04.416978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.416953 5272 generic.go:356] "Generic (PLEG): container finished" podID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerID="926d110a37ea6628c576b6d7d3b3776e330cba83305c5caf29057add9915e42a" exitCode=60 Nov 03 23:00:04.417094 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.417035 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" event={"ID":"38737a60-c402-4a2a-b52a-d765dcc36e1c","Type":"ContainerDied","Data":"926d110a37ea6628c576b6d7d3b3776e330cba83305c5caf29057add9915e42a"} Nov 03 23:00:04.417171 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.417090 5272 scope.go:122] "RemoveContainer" containerID="80985a8fedc16b96fc2d239dc51b9d556e6393966e5e39206652918170ae280a" Nov 03 23:00:04.417289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.417270 5272 scope.go:122] "RemoveContainer" containerID="926d110a37ea6628c576b6d7d3b3776e330cba83305c5caf29057add9915e42a" Nov 03 23:00:04.417620 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:04.417596 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp_odh-ai-gateway-infra(38737a60-c402-4a2a-b52a-d765dcc36e1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" Nov 03 23:00:04.419138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.419114 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" event={"ID":"5d767797-1338-49e6-b3ad-57fa05f249b5","Type":"ContainerStarted","Data":"3eeca975cbe9c311f98a15c7c107c7a7ed8f4bc1ce282c0d46d2ada7375d4d60"} Nov 03 23:00:04.421040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.421021 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" event={"ID":"09966415-3039-4b0f-8ab5-47f1d1e8cdf0","Type":"ContainerStarted","Data":"03724e8a9c5e2b21cc2485b8e034a329cd1f2882e96f47c697009c22c8f58b25"} Nov 03 23:00:04.422964 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.422941 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" event={"ID":"126243d7-7eaa-47db-b2da-615c6559e529","Type":"ContainerStarted","Data":"f215852a77f9644132311e452dc51092d73780fc186add68774f01efa5672b7a"} Nov 03 23:00:04.425012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.424989 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" event={"ID":"ed28ee9e-6798-4f37-8020-5325aa4c63cb","Type":"ContainerStarted","Data":"781da01aae8822362ce8775e29c6999736bc2cec50d2e799dc16e571b199cbb3"} Nov 03 23:00:04.432163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.427364 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" event={"ID":"8d6781d3-1e50-4b96-8874-2dc962a89a4f","Type":"ContainerStarted","Data":"bd0b7a635a7ebbcf4877e57a8de4b056217b9a4740845a335d9ff40ee3d91c3e"} Nov 03 23:00:04.432163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.430362 5272 generic.go:356] "Generic (PLEG): container finished" podID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerID="c1f3a919ac94aaf98d3075d3deefdce7f67da99e837316de2f19363c9332908f" exitCode=60 Nov 03 23:00:04.432163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.430427 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" event={"ID":"7d909e24-6172-4e2a-b04e-8a0c7c203a0f","Type":"ContainerDied","Data":"c1f3a919ac94aaf98d3075d3deefdce7f67da99e837316de2f19363c9332908f"} Nov 03 23:00:04.432163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.430703 5272 scope.go:122] "RemoveContainer" containerID="c1f3a919ac94aaf98d3075d3deefdce7f67da99e837316de2f19363c9332908f" Nov 03 23:00:04.432163 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:04.430962 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4_odh-ai-gateway-infra(7d909e24-6172-4e2a-b04e-8a0c7c203a0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" Nov 03 23:00:04.434889 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.434872 5272 scope.go:122] "RemoveContainer" containerID="df8c9de8d3b711ec092e9c431019dbf910410a42512a371379c27084a0f1afef" Nov 03 23:00:04.435013 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.434989 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerStarted","Data":"445f1d4644951c680b76eb407e961b41af514c8e5bd57cb59ecf7daa89739f5c"} Nov 03 23:00:04.436893 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.436871 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" event={"ID":"641c211c-d139-4894-ae2f-26e70a843973","Type":"ContainerStarted","Data":"993bbd477f68c37627e830956391282304b04c530949c03c4b0772343882c240"} Nov 03 23:00:04.480876 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:04.480817 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" podStartSLOduration=4.480786869 podStartE2EDuration="4.480786869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:04.480072133 +0000 UTC m=+2185.102380442" watchObservedRunningTime="2026-11-03 23:00:04.480786869 +0000 UTC m=+2185.103095179" Nov 03 23:00:05.447246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.447220 5272 scope.go:122] "RemoveContainer" containerID="c1f3a919ac94aaf98d3075d3deefdce7f67da99e837316de2f19363c9332908f" Nov 03 23:00:05.447678 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.447534 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4_odh-ai-gateway-infra(7d909e24-6172-4e2a-b04e-8a0c7c203a0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" Nov 03 23:00:05.448849 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.448823 5272 generic.go:356] "Generic (PLEG): container finished" podID="eec2ed53-af75-461b-872c-27f9b7bef027" containerID="445f1d4644951c680b76eb407e961b41af514c8e5bd57cb59ecf7daa89739f5c" exitCode=60 Nov 03 23:00:05.448984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.448905 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerDied","Data":"445f1d4644951c680b76eb407e961b41af514c8e5bd57cb59ecf7daa89739f5c"} Nov 03 23:00:05.448984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.448967 5272 scope.go:122] "RemoveContainer" containerID="7665648b1e8362d57df7029d3ef32a345ad479d70455729b2692b5b75c1c7709" Nov 03 23:00:05.449163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.449146 5272 scope.go:122] "RemoveContainer" containerID="445f1d4644951c680b76eb407e961b41af514c8e5bd57cb59ecf7daa89739f5c" Nov 03 23:00:05.449445 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.449421 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk_odh-ai-gateway-infra(eec2ed53-af75-461b-872c-27f9b7bef027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" Nov 03 23:00:05.450852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.450834 5272 generic.go:356] "Generic (PLEG): container finished" podID="641c211c-d139-4894-ae2f-26e70a843973" containerID="993bbd477f68c37627e830956391282304b04c530949c03c4b0772343882c240" exitCode=60 Nov 03 23:00:05.450939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.450910 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" event={"ID":"641c211c-d139-4894-ae2f-26e70a843973","Type":"ContainerDied","Data":"993bbd477f68c37627e830956391282304b04c530949c03c4b0772343882c240"} Nov 03 23:00:05.451126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.451106 5272 scope.go:122] "RemoveContainer" containerID="993bbd477f68c37627e830956391282304b04c530949c03c4b0772343882c240" Nov 03 23:00:05.451383 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.451358 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc_odh-ai-gateway-infra(641c211c-d139-4894-ae2f-26e70a843973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" podUID="641c211c-d139-4894-ae2f-26e70a843973" Nov 03 23:00:05.452715 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.452693 5272 generic.go:356] "Generic (PLEG): container finished" podID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerID="3f32e0eb89b802f85b03f4f4a0c12254df8c64bad0e749fbd979dadc799d98c8" exitCode=60 Nov 03 23:00:05.452829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.452761 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" event={"ID":"fc425062-2635-4d50-81fc-56bd9a9dab42","Type":"ContainerDied","Data":"3f32e0eb89b802f85b03f4f4a0c12254df8c64bad0e749fbd979dadc799d98c8"} Nov 03 23:00:05.452994 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.452979 5272 scope.go:122] "RemoveContainer" containerID="3f32e0eb89b802f85b03f4f4a0c12254df8c64bad0e749fbd979dadc799d98c8" Nov 03 23:00:05.453256 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.453226 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn_odh-ai-gateway-infra(fc425062-2635-4d50-81fc-56bd9a9dab42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" Nov 03 23:00:05.454621 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.454602 5272 scope.go:122] "RemoveContainer" containerID="926d110a37ea6628c576b6d7d3b3776e330cba83305c5caf29057add9915e42a" Nov 03 23:00:05.454886 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.454862 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp_odh-ai-gateway-infra(38737a60-c402-4a2a-b52a-d765dcc36e1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" Nov 03 23:00:05.456419 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.456309 5272 generic.go:356] "Generic (PLEG): container finished" podID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerID="3eeca975cbe9c311f98a15c7c107c7a7ed8f4bc1ce282c0d46d2ada7375d4d60" exitCode=60 Nov 03 23:00:05.456500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.456442 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" event={"ID":"5d767797-1338-49e6-b3ad-57fa05f249b5","Type":"ContainerDied","Data":"3eeca975cbe9c311f98a15c7c107c7a7ed8f4bc1ce282c0d46d2ada7375d4d60"} Nov 03 23:00:05.456644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.456626 5272 scope.go:122] "RemoveContainer" containerID="3eeca975cbe9c311f98a15c7c107c7a7ed8f4bc1ce282c0d46d2ada7375d4d60" Nov 03 23:00:05.456910 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.456885 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-dv2fb_odh-ai-gateway-infra(5d767797-1338-49e6-b3ad-57fa05f249b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" Nov 03 23:00:05.458612 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.458593 5272 generic.go:356] "Generic (PLEG): container finished" podID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerID="03724e8a9c5e2b21cc2485b8e034a329cd1f2882e96f47c697009c22c8f58b25" exitCode=60 Nov 03 23:00:05.458694 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.458619 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" event={"ID":"09966415-3039-4b0f-8ab5-47f1d1e8cdf0","Type":"ContainerDied","Data":"03724e8a9c5e2b21cc2485b8e034a329cd1f2882e96f47c697009c22c8f58b25"} Nov 03 23:00:05.458947 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.458923 5272 scope.go:122] "RemoveContainer" containerID="03724e8a9c5e2b21cc2485b8e034a329cd1f2882e96f47c697009c22c8f58b25" Nov 03 23:00:05.459232 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.459210 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94_odh-ai-gateway-infra(09966415-3039-4b0f-8ab5-47f1d1e8cdf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" Nov 03 23:00:05.460493 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.460475 5272 generic.go:356] "Generic (PLEG): container finished" podID="126243d7-7eaa-47db-b2da-615c6559e529" containerID="f215852a77f9644132311e452dc51092d73780fc186add68774f01efa5672b7a" exitCode=60 Nov 03 23:00:05.460577 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.460554 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" event={"ID":"126243d7-7eaa-47db-b2da-615c6559e529","Type":"ContainerDied","Data":"f215852a77f9644132311e452dc51092d73780fc186add68774f01efa5672b7a"} Nov 03 23:00:05.460731 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.460715 5272 scope.go:122] "RemoveContainer" containerID="f215852a77f9644132311e452dc51092d73780fc186add68774f01efa5672b7a" Nov 03 23:00:05.460965 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.460943 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8_odh-ai-gateway-infra(126243d7-7eaa-47db-b2da-615c6559e529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" podUID="126243d7-7eaa-47db-b2da-615c6559e529" Nov 03 23:00:05.462493 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.462473 5272 generic.go:356] "Generic (PLEG): container finished" podID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerID="781da01aae8822362ce8775e29c6999736bc2cec50d2e799dc16e571b199cbb3" exitCode=60 Nov 03 23:00:05.462576 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.462534 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" event={"ID":"ed28ee9e-6798-4f37-8020-5325aa4c63cb","Type":"ContainerDied","Data":"781da01aae8822362ce8775e29c6999736bc2cec50d2e799dc16e571b199cbb3"} Nov 03 23:00:05.462763 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.462747 5272 scope.go:122] "RemoveContainer" containerID="781da01aae8822362ce8775e29c6999736bc2cec50d2e799dc16e571b199cbb3" Nov 03 23:00:05.462993 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.462971 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw_odh-ai-gateway-infra(ed28ee9e-6798-4f37-8020-5325aa4c63cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" Nov 03 23:00:05.464210 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.464194 5272 scope.go:122] "RemoveContainer" containerID="d171fc5f314b6c85a1cf0735a949b9bead4bf7b325cdec63a256f890cf2ed7a7" Nov 03 23:00:05.464571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.464549 5272 generic.go:356] "Generic (PLEG): container finished" podID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerID="bd0b7a635a7ebbcf4877e57a8de4b056217b9a4740845a335d9ff40ee3d91c3e" exitCode=60 Nov 03 23:00:05.464673 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.464615 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" event={"ID":"8d6781d3-1e50-4b96-8874-2dc962a89a4f","Type":"ContainerDied","Data":"bd0b7a635a7ebbcf4877e57a8de4b056217b9a4740845a335d9ff40ee3d91c3e"} Nov 03 23:00:05.464881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.464863 5272 scope.go:122] "RemoveContainer" containerID="bd0b7a635a7ebbcf4877e57a8de4b056217b9a4740845a335d9ff40ee3d91c3e" Nov 03 23:00:05.465775 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:05.465092 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw_odh-ai-gateway-infra(8d6781d3-1e50-4b96-8874-2dc962a89a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" Nov 03 23:00:05.605599 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.605569 5272 scope.go:122] "RemoveContainer" containerID="6ebf8e288d03bbd738850203215c38e88d550cbbbe1f3cd911a70bf7ade65a6f" Nov 03 23:00:05.635585 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.635560 5272 scope.go:122] "RemoveContainer" containerID="dfcb2d2966da08f5f51b395aa70b855e1a817db2380fe4b179151a4295c1881e" Nov 03 23:00:05.650984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.650960 5272 scope.go:122] "RemoveContainer" containerID="32a47e38188f51e0f401e45f90cc13bf31a41206ec2fd1e4c843a00ba02a4f12" Nov 03 23:00:05.672963 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.672911 5272 scope.go:122] "RemoveContainer" containerID="dcb35cddbc4853e70acc38fcc8ac04ce859077c56c2a2693113ab835ef0c7f95" Nov 03 23:00:05.686659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.686629 5272 scope.go:122] "RemoveContainer" containerID="58f2728a87e592b384a0424c7e8a0e7e2296934331b33122448d31cf98cac618" Nov 03 23:00:05.692673 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.692642 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7"] Nov 03 23:00:05.704204 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.704185 5272 scope.go:122] "RemoveContainer" containerID="e6e719da2ba10b92f9542468cdb580f46dc43fb8de64bc6c46e3a942ada90496" Nov 03 23:00:05.804493 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.804454 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff"] Nov 03 23:00:05.878795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:05.878762 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94"] Nov 03 23:00:06.475628 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.475602 5272 scope.go:122] "RemoveContainer" containerID="781da01aae8822362ce8775e29c6999736bc2cec50d2e799dc16e571b199cbb3" Nov 03 23:00:06.476124 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.475944 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw_odh-ai-gateway-infra(ed28ee9e-6798-4f37-8020-5325aa4c63cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" Nov 03 23:00:06.477491 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.477465 5272 scope.go:122] "RemoveContainer" containerID="bd0b7a635a7ebbcf4877e57a8de4b056217b9a4740845a335d9ff40ee3d91c3e" Nov 03 23:00:06.477736 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.477711 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw_odh-ai-gateway-infra(8d6781d3-1e50-4b96-8874-2dc962a89a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" Nov 03 23:00:06.479238 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.479222 5272 scope.go:122] "RemoveContainer" containerID="445f1d4644951c680b76eb407e961b41af514c8e5bd57cb59ecf7daa89739f5c" Nov 03 23:00:06.479449 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.479426 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk_odh-ai-gateway-infra(eec2ed53-af75-461b-872c-27f9b7bef027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" Nov 03 23:00:06.480920 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.480890 5272 scope.go:122] "RemoveContainer" containerID="993bbd477f68c37627e830956391282304b04c530949c03c4b0772343882c240" Nov 03 23:00:06.481517 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.481234 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc_odh-ai-gateway-infra(641c211c-d139-4894-ae2f-26e70a843973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" podUID="641c211c-d139-4894-ae2f-26e70a843973" Nov 03 23:00:06.482954 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.482936 5272 scope.go:122] "RemoveContainer" containerID="3f32e0eb89b802f85b03f4f4a0c12254df8c64bad0e749fbd979dadc799d98c8" Nov 03 23:00:06.483153 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.483130 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn_odh-ai-gateway-infra(fc425062-2635-4d50-81fc-56bd9a9dab42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" Nov 03 23:00:06.484585 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.484564 5272 scope.go:122] "RemoveContainer" containerID="3eeca975cbe9c311f98a15c7c107c7a7ed8f4bc1ce282c0d46d2ada7375d4d60" Nov 03 23:00:06.484833 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.484796 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-dv2fb_odh-ai-gateway-infra(5d767797-1338-49e6-b3ad-57fa05f249b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" Nov 03 23:00:06.486631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.486613 5272 scope.go:122] "RemoveContainer" containerID="03724e8a9c5e2b21cc2485b8e034a329cd1f2882e96f47c697009c22c8f58b25" Nov 03 23:00:06.486818 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.486781 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94_odh-ai-gateway-infra(09966415-3039-4b0f-8ab5-47f1d1e8cdf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" Nov 03 23:00:06.488162 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.488147 5272 scope.go:122] "RemoveContainer" containerID="f215852a77f9644132311e452dc51092d73780fc186add68774f01efa5672b7a" Nov 03 23:00:06.488326 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:06.488310 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8_odh-ai-gateway-infra(126243d7-7eaa-47db-b2da-615c6559e529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" podUID="126243d7-7eaa-47db-b2da-615c6559e529" Nov 03 23:00:06.768904 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.768692 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx"] Nov 03 23:00:06.792826 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:06.790627 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2769cb-bm2kx"] Nov 03 23:00:07.512324 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:07.512287 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d5eb43-1d67-4bba-b9f3-e347d9176d98" path="/var/lib/kubelet/pods/69d5eb43-1d67-4bba-b9f3-e347d9176d98/volumes" Nov 03 23:00:07.834832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:07.826016 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:07.909938 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:07.909902 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-maas-api-service-ca\") pod \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " Nov 03 23:00:07.910127 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:07.910082 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-kube-api-access-9cfmw\" (UniqueName: \"kubernetes.io/projected/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-kube-api-access-9cfmw\") pod \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\" (UID: \"09966415-3039-4b0f-8ab5-47f1d1e8cdf0\") " Nov 03 23:00:07.910298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:07.910274 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-maas-api-service-ca" pod "09966415-3039-4b0f-8ab5-47f1d1e8cdf0" (UID: "09966415-3039-4b0f-8ab5-47f1d1e8cdf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:07.910618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:07.910587 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:07.912240 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:07.912209 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-kube-api-access-9cfmw" pod "09966415-3039-4b0f-8ab5-47f1d1e8cdf0" (UID: "09966415-3039-4b0f-8ab5-47f1d1e8cdf0"). InnerVolumeSpecName "kube-api-access-9cfmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:08.011868 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.011844 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cfmw\" (UniqueName: \"kubernetes.io/projected/09966415-3039-4b0f-8ab5-47f1d1e8cdf0-kube-api-access-9cfmw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:08.553446 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.553403 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" event={"ID":"09966415-3039-4b0f-8ab5-47f1d1e8cdf0","Type":"ContainerDied","Data":"02563bdf3354f4e23315939d87e7ef8e6c5714ade59a24dd71eb864a8b1fd49f"} Nov 03 23:00:08.553981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.553478 5272 scope.go:122] "RemoveContainer" containerID="03724e8a9c5e2b21cc2485b8e034a329cd1f2882e96f47c697009c22c8f58b25" Nov 03 23:00:08.553981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.553418 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94" Nov 03 23:00:08.705633 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.705602 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb"] Nov 03 23:00:08.713386 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.713350 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9674f5-v2npb"] Nov 03 23:00:08.716791 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.716765 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94"] Nov 03 23:00:08.722662 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:08.722636 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9674f5-29895780-nxq94"] Nov 03 23:00:09.496295 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:09.496260 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" path="/var/lib/kubelet/pods/09966415-3039-4b0f-8ab5-47f1d1e8cdf0/volumes" Nov 03 23:00:09.496719 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:09.496705 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a503b3-b09b-48d7-a396-2afe6dab2f0d" path="/var/lib/kubelet/pods/76a503b3-b09b-48d7-a396-2afe6dab2f0d/volumes" Nov 03 23:00:10.693318 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:10.693281 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" podUID="e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" containerName="payload-processing" containerID="cri-o://d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1" gracePeriod=25 Nov 03 23:00:10.805353 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:10.805316 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" podUID="9b745074-da33-4a98-9474-a09fa0ca0704" containerName="payload-pre-processing" containerID="cri-o://70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71" gracePeriod=25 Nov 03 23:00:10.997575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:10.997549 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 23:00:11.076683 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.076652 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-plugins-config-volume\") pod \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " Nov 03 23:00:11.076900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.076699 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-kube-api-access-gb2tc\" (UniqueName: \"kubernetes.io/projected/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-kube-api-access-gb2tc\") pod \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\" (UID: \"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9\") " Nov 03 23:00:11.077107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.077080 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-plugins-config-volume" pod "e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" (UID: "e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:11.078850 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.078826 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-kube-api-access-gb2tc" pod "e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" (UID: "e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9"). InnerVolumeSpecName "kube-api-access-gb2tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:11.099760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.099736 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 23:00:11.182880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.178359 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b745074-da33-4a98-9474-a09fa0ca0704-kube-api-access-5r664\" (UniqueName: \"kubernetes.io/projected/9b745074-da33-4a98-9474-a09fa0ca0704-kube-api-access-5r664\") pod \"9b745074-da33-4a98-9474-a09fa0ca0704\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " Nov 03 23:00:11.182880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.178420 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b745074-da33-4a98-9474-a09fa0ca0704-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b745074-da33-4a98-9474-a09fa0ca0704-plugins-config-volume\") pod \"9b745074-da33-4a98-9474-a09fa0ca0704\" (UID: \"9b745074-da33-4a98-9474-a09fa0ca0704\") " Nov 03 23:00:11.182880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.179950 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:11.182880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.179979 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb2tc\" (UniqueName: \"kubernetes.io/projected/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9-kube-api-access-gb2tc\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:11.182880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.180394 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b745074-da33-4a98-9474-a09fa0ca0704-plugins-config-volume" pod "9b745074-da33-4a98-9474-a09fa0ca0704" (UID: "9b745074-da33-4a98-9474-a09fa0ca0704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:11.188238 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.183757 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b745074-da33-4a98-9474-a09fa0ca0704-kube-api-access-5r664" pod "9b745074-da33-4a98-9474-a09fa0ca0704" (UID: "9b745074-da33-4a98-9474-a09fa0ca0704"). InnerVolumeSpecName "kube-api-access-5r664". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:11.281627 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.281531 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r664\" (UniqueName: \"kubernetes.io/projected/9b745074-da33-4a98-9474-a09fa0ca0704-kube-api-access-5r664\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:11.281627 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.281571 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b745074-da33-4a98-9474-a09fa0ca0704-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:11.381072 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.381038 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc"] Nov 03 23:00:11.383177 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.383128 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" containerID="cri-o://c35c0142876b35a0c5426c979892b44e66110233106dee9589df8c22ce58d2af" gracePeriod=30 Nov 03 23:00:11.386048 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.386024 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc"] Nov 03 23:00:11.386368 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.386340 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" containerID="cri-o://a32dc59afb658b1e58bc6e3028c200ba19fd9e0a0361024fd4794e6e92b9900f" gracePeriod=30 Nov 03 23:00:11.392794 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.392770 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd"] Nov 03 23:00:11.393017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.392990 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" containerID="cri-o://fd416274e0158b093fd2b030053fdccf4714444d39ff98f42b02e95f208fe711" gracePeriod=30 Nov 03 23:00:11.657287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.657246 5272 generic.go:356] "Generic (PLEG): container finished" podID="9b745074-da33-4a98-9474-a09fa0ca0704" containerID="70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71" exitCode=0 Nov 03 23:00:11.657512 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.657490 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" Nov 03 23:00:11.657613 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.657540 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" event={"ID":"9b745074-da33-4a98-9474-a09fa0ca0704","Type":"ContainerDied","Data":"70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71"} Nov 03 23:00:11.657669 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.657641 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff" event={"ID":"9b745074-da33-4a98-9474-a09fa0ca0704","Type":"ContainerDied","Data":"768625963f3f00b9d023947384dc40d7a120e05f9fd1f7da950bfc30dfe7b4dc"} Nov 03 23:00:11.657728 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.657687 5272 scope.go:122] "RemoveContainer" containerID="70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71" Nov 03 23:00:11.659771 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.659746 5272 generic.go:356] "Generic (PLEG): container finished" podID="e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" containerID="d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1" exitCode=0 Nov 03 23:00:11.659958 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.659828 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" event={"ID":"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9","Type":"ContainerDied","Data":"d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1"} Nov 03 23:00:11.659958 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.659854 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" Nov 03 23:00:11.659958 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.659874 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7" event={"ID":"e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9","Type":"ContainerDied","Data":"348504dcbaf4af5f40678bc436cb56946653ef7c3cc1e719c64022369bfdedbb"} Nov 03 23:00:11.669680 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.669658 5272 scope.go:122] "RemoveContainer" containerID="70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71" Nov 03 23:00:11.669981 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:11.669961 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71\": container with ID starting with 70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71 not found: ID does not exist" containerID="70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71" Nov 03 23:00:11.670074 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.669991 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71"} err="failed to get container status \"70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71\": rpc error: code = NotFound desc = could not find container \"70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71\": container with ID starting with 70277589663b6d8991379fb0f272dff67434b79b590368410dc824b405a89a71 not found: ID does not exist" Nov 03 23:00:11.670074 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.670014 5272 scope.go:122] "RemoveContainer" containerID="d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1" Nov 03 23:00:11.679467 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.679447 5272 scope.go:122] "RemoveContainer" containerID="d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1" Nov 03 23:00:11.679705 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:11.679686 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1\": container with ID starting with d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1 not found: ID does not exist" containerID="d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1" Nov 03 23:00:11.679765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.679714 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1"} err="failed to get container status \"d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1\": rpc error: code = NotFound desc = could not find container \"d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1\": container with ID starting with d5514d2e13c353af61cbeb4504c5361df40ad048b313aea0a4da42f2462a7bb1 not found: ID does not exist" Nov 03 23:00:11.898905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.898819 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7"] Nov 03 23:00:11.906488 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:11.906456 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6296b-95dc94cbb-k2wv7"] Nov 03 23:00:12.009016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:12.008985 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff"] Nov 03 23:00:12.013191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:12.013165 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6296b-b54fff875-sktff"] Nov 03 23:00:13.494347 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:13.494313 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b745074-da33-4a98-9474-a09fa0ca0704" path="/var/lib/kubelet/pods/9b745074-da33-4a98-9474-a09fa0ca0704/volumes" Nov 03 23:00:13.494797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:13.494765 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" path="/var/lib/kubelet/pods/e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9/volumes" Nov 03 23:00:14.486051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:14.486024 5272 scope.go:122] "RemoveContainer" containerID="926d110a37ea6628c576b6d7d3b3776e330cba83305c5caf29057add9915e42a" Nov 03 23:00:15.486335 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:15.486299 5272 scope.go:122] "RemoveContainer" containerID="bd0b7a635a7ebbcf4877e57a8de4b056217b9a4740845a335d9ff40ee3d91c3e" Nov 03 23:00:15.486850 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:15.486343 5272 scope.go:122] "RemoveContainer" containerID="c1f3a919ac94aaf98d3075d3deefdce7f67da99e837316de2f19363c9332908f" Nov 03 23:00:15.698859 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:15.698772 5272 generic.go:356] "Generic (PLEG): container finished" podID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerID="f4b16a12c5d2109609bb6016956c18823124e258325de6a1b6f810dbf5029ab0" exitCode=60 Nov 03 23:00:15.699005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:15.698852 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" event={"ID":"38737a60-c402-4a2a-b52a-d765dcc36e1c","Type":"ContainerDied","Data":"f4b16a12c5d2109609bb6016956c18823124e258325de6a1b6f810dbf5029ab0"} Nov 03 23:00:15.699005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:15.698916 5272 scope.go:122] "RemoveContainer" containerID="926d110a37ea6628c576b6d7d3b3776e330cba83305c5caf29057add9915e42a" Nov 03 23:00:15.699305 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:15.699286 5272 scope.go:122] "RemoveContainer" containerID="f4b16a12c5d2109609bb6016956c18823124e258325de6a1b6f810dbf5029ab0" Nov 03 23:00:15.699661 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:15.699637 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp_odh-ai-gateway-infra(38737a60-c402-4a2a-b52a-d765dcc36e1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" Nov 03 23:00:16.486249 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.486223 5272 scope.go:122] "RemoveContainer" containerID="445f1d4644951c680b76eb407e961b41af514c8e5bd57cb59ecf7daa89739f5c" Nov 03 23:00:16.720904 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.720878 5272 generic.go:356] "Generic (PLEG): container finished" podID="f32b4f71-9f49-4f92-b766-88ce76449123" containerID="a32dc59afb658b1e58bc6e3028c200ba19fd9e0a0361024fd4794e6e92b9900f" exitCode=0 Nov 03 23:00:16.721265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.720950 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" event={"ID":"f32b4f71-9f49-4f92-b766-88ce76449123","Type":"ContainerDied","Data":"a32dc59afb658b1e58bc6e3028c200ba19fd9e0a0361024fd4794e6e92b9900f"} Nov 03 23:00:16.721265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.721010 5272 scope.go:122] "RemoveContainer" containerID="5aab74f971329a16fb7ee278f4f3db2bfbea96d0eddcb514313eedc9eab7986d" Nov 03 23:00:16.723176 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.723156 5272 generic.go:356] "Generic (PLEG): container finished" podID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerID="dbc9f5c854c90f04212e3e7852b0f6110465cddde5f28fc020e59e27eb1a9890" exitCode=60 Nov 03 23:00:16.723282 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.723232 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" event={"ID":"8d6781d3-1e50-4b96-8874-2dc962a89a4f","Type":"ContainerDied","Data":"dbc9f5c854c90f04212e3e7852b0f6110465cddde5f28fc020e59e27eb1a9890"} Nov 03 23:00:16.723728 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.723701 5272 scope.go:122] "RemoveContainer" containerID="dbc9f5c854c90f04212e3e7852b0f6110465cddde5f28fc020e59e27eb1a9890" Nov 03 23:00:16.724207 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:16.724161 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw_odh-ai-gateway-infra(8d6781d3-1e50-4b96-8874-2dc962a89a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" Nov 03 23:00:16.725917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.725894 5272 generic.go:356] "Generic (PLEG): container finished" podID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerID="5885aebe1c296a98899b97513688a64582d94a892dbb34615bc8aabd5266fcb4" exitCode=60 Nov 03 23:00:16.726010 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.725979 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" event={"ID":"7d909e24-6172-4e2a-b04e-8a0c7c203a0f","Type":"ContainerDied","Data":"5885aebe1c296a98899b97513688a64582d94a892dbb34615bc8aabd5266fcb4"} Nov 03 23:00:16.726397 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.726378 5272 scope.go:122] "RemoveContainer" containerID="5885aebe1c296a98899b97513688a64582d94a892dbb34615bc8aabd5266fcb4" Nov 03 23:00:16.726728 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:16.726706 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4_odh-ai-gateway-infra(7d909e24-6172-4e2a-b04e-8a0c7c203a0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" Nov 03 23:00:16.727530 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.727510 5272 generic.go:356] "Generic (PLEG): container finished" podID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerID="fd416274e0158b093fd2b030053fdccf4714444d39ff98f42b02e95f208fe711" exitCode=0 Nov 03 23:00:16.727618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.727580 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" event={"ID":"e5a650cb-cc75-4d8e-b4bc-5772595e45bb","Type":"ContainerDied","Data":"fd416274e0158b093fd2b030053fdccf4714444d39ff98f42b02e95f208fe711"} Nov 03 23:00:16.730671 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.730652 5272 generic.go:356] "Generic (PLEG): container finished" podID="600c464b-672f-46bd-8122-0b44bb476a9a" containerID="c35c0142876b35a0c5426c979892b44e66110233106dee9589df8c22ce58d2af" exitCode=0 Nov 03 23:00:16.730751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.730698 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" event={"ID":"600c464b-672f-46bd-8122-0b44bb476a9a","Type":"ContainerDied","Data":"c35c0142876b35a0c5426c979892b44e66110233106dee9589df8c22ce58d2af"} Nov 03 23:00:16.730751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.730725 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" event={"ID":"600c464b-672f-46bd-8122-0b44bb476a9a","Type":"ContainerDied","Data":"7c8730139912c2734b113ef0d9a8e9f7694bdabbddb6d1c3bd3f26592bfcd488"} Nov 03 23:00:16.730751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.730741 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8730139912c2734b113ef0d9a8e9f7694bdabbddb6d1c3bd3f26592bfcd488" Nov 03 23:00:16.750307 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.750291 5272 scope.go:122] "RemoveContainer" containerID="bd0b7a635a7ebbcf4877e57a8de4b056217b9a4740845a335d9ff40ee3d91c3e" Nov 03 23:00:16.761494 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.761472 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 23:00:16.765932 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.765914 5272 scope.go:122] "RemoveContainer" containerID="c1f3a919ac94aaf98d3075d3deefdce7f67da99e837316de2f19363c9332908f" Nov 03 23:00:16.769594 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.769577 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 23:00:16.772958 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.772942 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 23:00:16.822561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.822532 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp"] Nov 03 23:00:16.871263 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871231 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-certs\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.871418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871278 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-podinfo\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.871418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871297 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-data\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.871418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871322 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-envoy\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.871418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871351 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-kube-api-access-p777m\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-kube-api-access-p777m\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.871418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871367 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-istio-token\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-istio-token\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.871418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871398 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32b4f71-9f49-4f92-b766-88ce76449123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32b4f71-9f49-4f92-b766-88ce76449123-istiod-ca-cert\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871485 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-socket\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871579 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istiod-ca-cert\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871628 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-envoy\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871651 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-token\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-token\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871698 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-credential-socket\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871722 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-istio-token\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-istio-token\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871739 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-certs" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871782 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-credential-socket\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871832 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-data\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871863 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-socket\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.871905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871904 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600c464b-672f-46bd-8122-0b44bb476a9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600c464b-672f-46bd-8122-0b44bb476a9a-istiod-ca-cert\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871929 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-certs\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871979 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-socket\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.871979 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-data" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872016 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/600c464b-672f-46bd-8122-0b44bb476a9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600c464b-672f-46bd-8122-0b44bb476a9a-istio-podinfo\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872071 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-kube-api-access-z57p2\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-kube-api-access-z57p2\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872138 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-data\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872172 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-credential-socket\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872198 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-kube-api-access-hb4mv\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-kube-api-access-hb4mv\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872196 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-credential-socket" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872277 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-certs\") pod \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\" (UID: \"e5a650cb-cc75-4d8e-b4bc-5772595e45bb\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872322 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-envoy\") pod \"600c464b-672f-46bd-8122-0b44bb476a9a\" (UID: \"600c464b-672f-46bd-8122-0b44bb476a9a\") " Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872359 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istiod-ca-cert" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:16.872503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872374 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f32b4f71-9f49-4f92-b766-88ce76449123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32b4f71-9f49-4f92-b766-88ce76449123-istio-podinfo\") pod \"f32b4f71-9f49-4f92-b766-88ce76449123\" (UID: \"f32b4f71-9f49-4f92-b766-88ce76449123\") " Nov 03 23:00:16.873108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872600 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-socket" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.873108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.872760 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32b4f71-9f49-4f92-b766-88ce76449123-istiod-ca-cert" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:16.873108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.873061 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-socket" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.873276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.873250 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-socket" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.875346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.875085 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-data" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.875346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.875106 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-certs" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.878774 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.878594 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-certs" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.879154 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.878960 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-credential-socket" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.879154 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879052 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-kube-api-access-p777m" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "kube-api-access-p777m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.879404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879358 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-data" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.879837 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879560 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600c464b-672f-46bd-8122-0b44bb476a9a-istiod-ca-cert" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:16.879837 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879640 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-credential-socket" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.879992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879941 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.879992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879962 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.879992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879976 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p777m\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-kube-api-access-p777m\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.879992 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32b4f71-9f49-4f92-b766-88ce76449123-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880006 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880021 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880035 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880048 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880061 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880074 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880086 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600c464b-672f-46bd-8122-0b44bb476a9a-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880099 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880112 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880125 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880139 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.880742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.880152 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.881723 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.881671 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f32b4f71-9f49-4f92-b766-88ce76449123-istio-podinfo" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:16.881723 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.881698 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-token" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.882021 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.881978 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-kube-api-access-hb4mv" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "kube-api-access-hb4mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.882021 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.881992 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-envoy" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.882021 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.881993 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-podinfo" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:16.882204 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.882090 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-envoy" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.882279 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.882258 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-kube-api-access-z57p2" pod "e5a650cb-cc75-4d8e-b4bc-5772595e45bb" (UID: "e5a650cb-cc75-4d8e-b4bc-5772595e45bb"). InnerVolumeSpecName "kube-api-access-z57p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.882347 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.882286 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-envoy" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:16.882416 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.882400 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-istio-token" pod "f32b4f71-9f49-4f92-b766-88ce76449123" (UID: "f32b4f71-9f49-4f92-b766-88ce76449123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.882495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.882469 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-istio-token" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:16.887227 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.885240 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/600c464b-672f-46bd-8122-0b44bb476a9a-istio-podinfo" pod "600c464b-672f-46bd-8122-0b44bb476a9a" (UID: "600c464b-672f-46bd-8122-0b44bb476a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:16.981443 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981414 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981443 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981443 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32b4f71-9f49-4f92-b766-88ce76449123-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981457 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32b4f71-9f49-4f92-b766-88ce76449123-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981471 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981480 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981488 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981496 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600c464b-672f-46bd-8122-0b44bb476a9a-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981505 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z57p2\" (UniqueName: \"kubernetes.io/projected/e5a650cb-cc75-4d8e-b4bc-5772595e45bb-kube-api-access-z57p2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981514 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4mv\" (UniqueName: \"kubernetes.io/projected/600c464b-672f-46bd-8122-0b44bb476a9a-kube-api-access-hb4mv\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981523 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600c464b-672f-46bd-8122-0b44bb476a9a-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.981590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.981534 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32b4f71-9f49-4f92-b766-88ce76449123-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:16.986661 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:16.986635 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:17.082324 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.082300 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38737a60-c402-4a2a-b52a-d765dcc36e1c-kube-api-access-dlrtb\" (UniqueName: \"kubernetes.io/projected/38737a60-c402-4a2a-b52a-d765dcc36e1c-kube-api-access-dlrtb\") pod \"38737a60-c402-4a2a-b52a-d765dcc36e1c\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " Nov 03 23:00:17.082480 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.082466 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38737a60-c402-4a2a-b52a-d765dcc36e1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38737a60-c402-4a2a-b52a-d765dcc36e1c-maas-api-service-ca\") pod \"38737a60-c402-4a2a-b52a-d765dcc36e1c\" (UID: \"38737a60-c402-4a2a-b52a-d765dcc36e1c\") " Nov 03 23:00:17.082846 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.082797 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38737a60-c402-4a2a-b52a-d765dcc36e1c-maas-api-service-ca" pod "38737a60-c402-4a2a-b52a-d765dcc36e1c" (UID: "38737a60-c402-4a2a-b52a-d765dcc36e1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:17.084170 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.084153 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38737a60-c402-4a2a-b52a-d765dcc36e1c-kube-api-access-dlrtb" pod "38737a60-c402-4a2a-b52a-d765dcc36e1c" (UID: "38737a60-c402-4a2a-b52a-d765dcc36e1c"). InnerVolumeSpecName "kube-api-access-dlrtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:17.184285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.184251 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrtb\" (UniqueName: \"kubernetes.io/projected/38737a60-c402-4a2a-b52a-d765dcc36e1c-kube-api-access-dlrtb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:17.184285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.184279 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38737a60-c402-4a2a-b52a-d765dcc36e1c-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:17.486510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.486475 5272 scope.go:122] "RemoveContainer" containerID="781da01aae8822362ce8775e29c6999736bc2cec50d2e799dc16e571b199cbb3" Nov 03 23:00:17.486702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.486598 5272 scope.go:122] "RemoveContainer" containerID="f215852a77f9644132311e452dc51092d73780fc186add68774f01efa5672b7a" Nov 03 23:00:17.486790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.486732 5272 scope.go:122] "RemoveContainer" containerID="3f32e0eb89b802f85b03f4f4a0c12254df8c64bad0e749fbd979dadc799d98c8" Nov 03 23:00:17.739400 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.739323 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" Nov 03 23:00:17.739400 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.739355 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp" event={"ID":"38737a60-c402-4a2a-b52a-d765dcc36e1c","Type":"ContainerDied","Data":"bff65f6665a8e8105f05440bfebfa84fb6d8221273a04b7d23daacf688387d49"} Nov 03 23:00:17.739971 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.739428 5272 scope.go:122] "RemoveContainer" containerID="f4b16a12c5d2109609bb6016956c18823124e258325de6a1b6f810dbf5029ab0" Nov 03 23:00:17.741081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.741060 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" Nov 03 23:00:17.741239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.741086 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc" event={"ID":"f32b4f71-9f49-4f92-b766-88ce76449123","Type":"ContainerDied","Data":"53731fff8c6664aba2d5e39082dffa432b7b4e8025b722bf674f2e50142d7282"} Nov 03 23:00:17.747040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.747018 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerStarted","Data":"d04570f4e6d625b0fe14d2e8c863954c45d836266eda2d75ff3c0eb9f9cb880e"} Nov 03 23:00:17.748499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.748480 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" Nov 03 23:00:17.748499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.748495 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd" event={"ID":"e5a650cb-cc75-4d8e-b4bc-5772595e45bb","Type":"ContainerDied","Data":"933a315ca2221ea5a7e306cd7616655b57f73026ff31117f2f6a677e501e790c"} Nov 03 23:00:17.748684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.748483 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc" Nov 03 23:00:17.811973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.811950 5272 scope.go:122] "RemoveContainer" containerID="a32dc59afb658b1e58bc6e3028c200ba19fd9e0a0361024fd4794e6e92b9900f" Nov 03 23:00:17.822907 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.822888 5272 scope.go:122] "RemoveContainer" containerID="fd416274e0158b093fd2b030053fdccf4714444d39ff98f42b02e95f208fe711" Nov 03 23:00:17.848051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:17.848017 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw"] Nov 03 23:00:18.061693 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.061667 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp"] Nov 03 23:00:18.063583 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.063560 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18c03a-29895780-zj9kp"] Nov 03 23:00:18.080363 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.080345 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:18.164546 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.164507 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc"] Nov 03 23:00:18.171090 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.171030 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-kh8pc"] Nov 03 23:00:18.174433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.174403 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4"] Nov 03 23:00:18.197470 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.197433 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6781d3-1e50-4b96-8874-2dc962a89a4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6781d3-1e50-4b96-8874-2dc962a89a4f-maas-api-service-ca\") pod \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " Nov 03 23:00:18.197658 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.197584 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6781d3-1e50-4b96-8874-2dc962a89a4f-kube-api-access-d5fj4\" (UniqueName: \"kubernetes.io/projected/8d6781d3-1e50-4b96-8874-2dc962a89a4f-kube-api-access-d5fj4\") pod \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\" (UID: \"8d6781d3-1e50-4b96-8874-2dc962a89a4f\") " Nov 03 23:00:18.197885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.197855 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6781d3-1e50-4b96-8874-2dc962a89a4f-maas-api-service-ca" pod "8d6781d3-1e50-4b96-8874-2dc962a89a4f" (UID: "8d6781d3-1e50-4b96-8874-2dc962a89a4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:18.198289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.198198 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d6781d3-1e50-4b96-8874-2dc962a89a4f-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:18.200005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.199976 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6781d3-1e50-4b96-8874-2dc962a89a4f-kube-api-access-d5fj4" pod "8d6781d3-1e50-4b96-8874-2dc962a89a4f" (UID: "8d6781d3-1e50-4b96-8874-2dc962a89a4f"). InnerVolumeSpecName "kube-api-access-d5fj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:18.313528 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.306594 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5fj4\" (UniqueName: \"kubernetes.io/projected/8d6781d3-1e50-4b96-8874-2dc962a89a4f-kube-api-access-d5fj4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:18.372841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.372784 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:18.458345 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.458293 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc"] Nov 03 23:00:18.465693 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.465670 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-vrdvc"] Nov 03 23:00:18.508755 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.508733 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-kube-api-access-fzf48\" (UniqueName: \"kubernetes.io/projected/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-kube-api-access-fzf48\") pod \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " Nov 03 23:00:18.508909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.508853 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-maas-api-service-ca\") pod \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\" (UID: \"7d909e24-6172-4e2a-b04e-8a0c7c203a0f\") " Nov 03 23:00:18.509184 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.509160 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-maas-api-service-ca" pod "7d909e24-6172-4e2a-b04e-8a0c7c203a0f" (UID: "7d909e24-6172-4e2a-b04e-8a0c7c203a0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:18.509427 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.509412 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:18.510756 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.510733 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-kube-api-access-fzf48" pod "7d909e24-6172-4e2a-b04e-8a0c7c203a0f" (UID: "7d909e24-6172-4e2a-b04e-8a0c7c203a0f"). InnerVolumeSpecName "kube-api-access-fzf48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:18.610978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.610893 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzf48\" (UniqueName: \"kubernetes.io/projected/7d909e24-6172-4e2a-b04e-8a0c7c203a0f-kube-api-access-fzf48\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:18.677333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.677303 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd"] Nov 03 23:00:18.682740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.682712 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12450a06-openshift-default-7576d969f7-t67gd"] Nov 03 23:00:18.758323 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.758286 5272 generic.go:356] "Generic (PLEG): container finished" podID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerID="b80b6fa5a5d8b9a8ec72f0fe34cc985fed5b55888c15e9ede234bc2910e1c9f3" exitCode=60 Nov 03 23:00:18.758745 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.758372 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" event={"ID":"ed28ee9e-6798-4f37-8020-5325aa4c63cb","Type":"ContainerDied","Data":"b80b6fa5a5d8b9a8ec72f0fe34cc985fed5b55888c15e9ede234bc2910e1c9f3"} Nov 03 23:00:18.758745 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.758438 5272 scope.go:122] "RemoveContainer" containerID="781da01aae8822362ce8775e29c6999736bc2cec50d2e799dc16e571b199cbb3" Nov 03 23:00:18.758920 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.758855 5272 scope.go:122] "RemoveContainer" containerID="b80b6fa5a5d8b9a8ec72f0fe34cc985fed5b55888c15e9ede234bc2910e1c9f3" Nov 03 23:00:18.759282 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:18.759247 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw_odh-ai-gateway-infra(ed28ee9e-6798-4f37-8020-5325aa4c63cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" Nov 03 23:00:18.760959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.760937 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" Nov 03 23:00:18.761070 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.760964 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw" event={"ID":"8d6781d3-1e50-4b96-8874-2dc962a89a4f","Type":"ContainerDied","Data":"4ae81754972267163bfd5a0f07841e315e36a986a100bf7f2beca2b5e04e689a"} Nov 03 23:00:18.762454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.762432 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" Nov 03 23:00:18.762544 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.762434 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4" event={"ID":"7d909e24-6172-4e2a-b04e-8a0c7c203a0f","Type":"ContainerDied","Data":"edd321200a9b089c5ae9a453d3906c97a60ffee9d89bd5eabf5f714777f4ef23"} Nov 03 23:00:18.764789 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.764767 5272 generic.go:356] "Generic (PLEG): container finished" podID="eec2ed53-af75-461b-872c-27f9b7bef027" containerID="d04570f4e6d625b0fe14d2e8c863954c45d836266eda2d75ff3c0eb9f9cb880e" exitCode=60 Nov 03 23:00:18.764914 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.764839 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerDied","Data":"d04570f4e6d625b0fe14d2e8c863954c45d836266eda2d75ff3c0eb9f9cb880e"} Nov 03 23:00:18.765194 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.765177 5272 scope.go:122] "RemoveContainer" containerID="d04570f4e6d625b0fe14d2e8c863954c45d836266eda2d75ff3c0eb9f9cb880e" Nov 03 23:00:18.765448 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:18.765428 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk_odh-ai-gateway-infra(eec2ed53-af75-461b-872c-27f9b7bef027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" Nov 03 23:00:18.768587 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.768561 5272 generic.go:356] "Generic (PLEG): container finished" podID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerID="b6f94c7e3db72d0eb80f9f2debafabce536137ada7915e411bc5437c1b4825bb" exitCode=60 Nov 03 23:00:18.768685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.768637 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" event={"ID":"fc425062-2635-4d50-81fc-56bd9a9dab42","Type":"ContainerDied","Data":"b6f94c7e3db72d0eb80f9f2debafabce536137ada7915e411bc5437c1b4825bb"} Nov 03 23:00:18.769045 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.769026 5272 scope.go:122] "RemoveContainer" containerID="b6f94c7e3db72d0eb80f9f2debafabce536137ada7915e411bc5437c1b4825bb" Nov 03 23:00:18.769309 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:18.769281 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn_odh-ai-gateway-infra(fc425062-2635-4d50-81fc-56bd9a9dab42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" Nov 03 23:00:18.773473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.773455 5272 generic.go:356] "Generic (PLEG): container finished" podID="126243d7-7eaa-47db-b2da-615c6559e529" containerID="0aa07e35fff7aaf47835cff83592009490d30aa0e4baec598742029854b0f04a" exitCode=60 Nov 03 23:00:18.773567 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.773483 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" event={"ID":"126243d7-7eaa-47db-b2da-615c6559e529","Type":"ContainerDied","Data":"0aa07e35fff7aaf47835cff83592009490d30aa0e4baec598742029854b0f04a"} Nov 03 23:00:18.773745 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.773732 5272 scope.go:122] "RemoveContainer" containerID="0aa07e35fff7aaf47835cff83592009490d30aa0e4baec598742029854b0f04a" Nov 03 23:00:18.773982 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:18.773961 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8_odh-ai-gateway-infra(126243d7-7eaa-47db-b2da-615c6559e529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" podUID="126243d7-7eaa-47db-b2da-615c6559e529" Nov 03 23:00:18.775596 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.775581 5272 scope.go:122] "RemoveContainer" containerID="dbc9f5c854c90f04212e3e7852b0f6110465cddde5f28fc020e59e27eb1a9890" Nov 03 23:00:18.784404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.784387 5272 scope.go:122] "RemoveContainer" containerID="5885aebe1c296a98899b97513688a64582d94a892dbb34615bc8aabd5266fcb4" Nov 03 23:00:18.796548 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.796516 5272 scope.go:122] "RemoveContainer" containerID="445f1d4644951c680b76eb407e961b41af514c8e5bd57cb59ecf7daa89739f5c" Nov 03 23:00:18.813737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.813717 5272 scope.go:122] "RemoveContainer" containerID="3f32e0eb89b802f85b03f4f4a0c12254df8c64bad0e749fbd979dadc799d98c8" Nov 03 23:00:18.840597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.840568 5272 scope.go:122] "RemoveContainer" containerID="f215852a77f9644132311e452dc51092d73780fc186add68774f01efa5672b7a" Nov 03 23:00:18.914522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:18.914453 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk"] Nov 03 23:00:19.124340 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.124305 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4"] Nov 03 23:00:19.131089 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.131066 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-545b5d-29895780-b5vh4"] Nov 03 23:00:19.342846 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.342775 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw"] Nov 03 23:00:19.347083 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.347055 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9b4383-29895780-zdwmw"] Nov 03 23:00:19.490248 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.490219 5272 scope.go:122] "RemoveContainer" containerID="993bbd477f68c37627e830956391282304b04c530949c03c4b0772343882c240" Nov 03 23:00:19.490444 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.490381 5272 scope.go:122] "RemoveContainer" containerID="3eeca975cbe9c311f98a15c7c107c7a7ed8f4bc1ce282c0d46d2ada7375d4d60" Nov 03 23:00:19.495336 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.495310 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" path="/var/lib/kubelet/pods/38737a60-c402-4a2a-b52a-d765dcc36e1c/volumes" Nov 03 23:00:19.495865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.495846 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" path="/var/lib/kubelet/pods/600c464b-672f-46bd-8122-0b44bb476a9a/volumes" Nov 03 23:00:19.496235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.496219 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" path="/var/lib/kubelet/pods/7d909e24-6172-4e2a-b04e-8a0c7c203a0f/volumes" Nov 03 23:00:19.496948 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.496930 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" path="/var/lib/kubelet/pods/8d6781d3-1e50-4b96-8874-2dc962a89a4f/volumes" Nov 03 23:00:19.497369 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.497352 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" path="/var/lib/kubelet/pods/e5a650cb-cc75-4d8e-b4bc-5772595e45bb/volumes" Nov 03 23:00:19.497763 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.497748 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" path="/var/lib/kubelet/pods/f32b4f71-9f49-4f92-b766-88ce76449123/volumes" Nov 03 23:00:19.928187 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:19.928150 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn"] Nov 03 23:00:20.033137 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.033106 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8"] Nov 03 23:00:20.096718 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.096691 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:20.167420 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.167392 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:20.186831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.186784 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:20.232706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.232683 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec2ed53-af75-461b-872c-27f9b7bef027-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec2ed53-af75-461b-872c-27f9b7bef027-maas-api-service-ca\") pod \"eec2ed53-af75-461b-872c-27f9b7bef027\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " Nov 03 23:00:20.232864 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.232789 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec2ed53-af75-461b-872c-27f9b7bef027-kube-api-access-nfsf6\" (UniqueName: \"kubernetes.io/projected/eec2ed53-af75-461b-872c-27f9b7bef027-kube-api-access-nfsf6\") pod \"eec2ed53-af75-461b-872c-27f9b7bef027\" (UID: \"eec2ed53-af75-461b-872c-27f9b7bef027\") " Nov 03 23:00:20.233062 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.233038 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec2ed53-af75-461b-872c-27f9b7bef027-maas-api-service-ca" pod "eec2ed53-af75-461b-872c-27f9b7bef027" (UID: "eec2ed53-af75-461b-872c-27f9b7bef027"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.233180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.233165 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec2ed53-af75-461b-872c-27f9b7bef027-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.234699 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.234680 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec2ed53-af75-461b-872c-27f9b7bef027-kube-api-access-nfsf6" pod "eec2ed53-af75-461b-872c-27f9b7bef027" (UID: "eec2ed53-af75-461b-872c-27f9b7bef027"). InnerVolumeSpecName "kube-api-access-nfsf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.251249 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.251225 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw"] Nov 03 23:00:20.334527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.334494 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc425062-2635-4d50-81fc-56bd9a9dab42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc425062-2635-4d50-81fc-56bd9a9dab42-maas-api-service-ca\") pod \"fc425062-2635-4d50-81fc-56bd9a9dab42\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " Nov 03 23:00:20.334681 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.334593 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126243d7-7eaa-47db-b2da-615c6559e529-kube-api-access-9v9lk\" (UniqueName: \"kubernetes.io/projected/126243d7-7eaa-47db-b2da-615c6559e529-kube-api-access-9v9lk\") pod \"126243d7-7eaa-47db-b2da-615c6559e529\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " Nov 03 23:00:20.334681 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.334619 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc425062-2635-4d50-81fc-56bd9a9dab42-kube-api-access-9b8nn\" (UniqueName: \"kubernetes.io/projected/fc425062-2635-4d50-81fc-56bd9a9dab42-kube-api-access-9b8nn\") pod \"fc425062-2635-4d50-81fc-56bd9a9dab42\" (UID: \"fc425062-2635-4d50-81fc-56bd9a9dab42\") " Nov 03 23:00:20.334681 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.334643 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126243d7-7eaa-47db-b2da-615c6559e529-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126243d7-7eaa-47db-b2da-615c6559e529-maas-api-service-ca\") pod \"126243d7-7eaa-47db-b2da-615c6559e529\" (UID: \"126243d7-7eaa-47db-b2da-615c6559e529\") " Nov 03 23:00:20.334992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.334962 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc425062-2635-4d50-81fc-56bd9a9dab42-maas-api-service-ca" pod "fc425062-2635-4d50-81fc-56bd9a9dab42" (UID: "fc425062-2635-4d50-81fc-56bd9a9dab42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.335418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.335392 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc425062-2635-4d50-81fc-56bd9a9dab42-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.335510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.335425 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfsf6\" (UniqueName: \"kubernetes.io/projected/eec2ed53-af75-461b-872c-27f9b7bef027-kube-api-access-nfsf6\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.335510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.335428 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126243d7-7eaa-47db-b2da-615c6559e529-maas-api-service-ca" pod "126243d7-7eaa-47db-b2da-615c6559e529" (UID: "126243d7-7eaa-47db-b2da-615c6559e529"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.337777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.337746 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc425062-2635-4d50-81fc-56bd9a9dab42-kube-api-access-9b8nn" pod "fc425062-2635-4d50-81fc-56bd9a9dab42" (UID: "fc425062-2635-4d50-81fc-56bd9a9dab42"). InnerVolumeSpecName "kube-api-access-9b8nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.337916 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.337885 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126243d7-7eaa-47db-b2da-615c6559e529-kube-api-access-9v9lk" pod "126243d7-7eaa-47db-b2da-615c6559e529" (UID: "126243d7-7eaa-47db-b2da-615c6559e529"). InnerVolumeSpecName "kube-api-access-9v9lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.389491 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.389467 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:20.436760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.436735 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9lk\" (UniqueName: \"kubernetes.io/projected/126243d7-7eaa-47db-b2da-615c6559e529-kube-api-access-9v9lk\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.436760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.436759 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b8nn\" (UniqueName: \"kubernetes.io/projected/fc425062-2635-4d50-81fc-56bd9a9dab42-kube-api-access-9b8nn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.436942 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.436768 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126243d7-7eaa-47db-b2da-615c6559e529-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.538198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.538171 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed28ee9e-6798-4f37-8020-5325aa4c63cb-kube-api-access-77zjp\" (UniqueName: \"kubernetes.io/projected/ed28ee9e-6798-4f37-8020-5325aa4c63cb-kube-api-access-77zjp\") pod \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " Nov 03 23:00:20.538391 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.538317 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed28ee9e-6798-4f37-8020-5325aa4c63cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed28ee9e-6798-4f37-8020-5325aa4c63cb-maas-api-service-ca\") pod \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\" (UID: \"ed28ee9e-6798-4f37-8020-5325aa4c63cb\") " Nov 03 23:00:20.538652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.538629 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed28ee9e-6798-4f37-8020-5325aa4c63cb-maas-api-service-ca" pod "ed28ee9e-6798-4f37-8020-5325aa4c63cb" (UID: "ed28ee9e-6798-4f37-8020-5325aa4c63cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.540588 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.540565 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed28ee9e-6798-4f37-8020-5325aa4c63cb-kube-api-access-77zjp" pod "ed28ee9e-6798-4f37-8020-5325aa4c63cb" (UID: "ed28ee9e-6798-4f37-8020-5325aa4c63cb"). InnerVolumeSpecName "kube-api-access-77zjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.639736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.639704 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed28ee9e-6798-4f37-8020-5325aa4c63cb-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.639736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.639731 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zjp\" (UniqueName: \"kubernetes.io/projected/ed28ee9e-6798-4f37-8020-5325aa4c63cb-kube-api-access-77zjp\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.798841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.798795 5272 generic.go:356] "Generic (PLEG): container finished" podID="641c211c-d139-4894-ae2f-26e70a843973" containerID="d0ef38842d46c96923cea6f6dc768f72133f1bb08252dc261d489d2bc54e220d" exitCode=60 Nov 03 23:00:20.799027 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.798872 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" event={"ID":"641c211c-d139-4894-ae2f-26e70a843973","Type":"ContainerDied","Data":"d0ef38842d46c96923cea6f6dc768f72133f1bb08252dc261d489d2bc54e220d"} Nov 03 23:00:20.799027 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.798943 5272 scope.go:122] "RemoveContainer" containerID="993bbd477f68c37627e830956391282304b04c530949c03c4b0772343882c240" Nov 03 23:00:20.799371 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.799346 5272 scope.go:122] "RemoveContainer" containerID="d0ef38842d46c96923cea6f6dc768f72133f1bb08252dc261d489d2bc54e220d" Nov 03 23:00:20.799717 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:20.799695 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc_odh-ai-gateway-infra(641c211c-d139-4894-ae2f-26e70a843973)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" podUID="641c211c-d139-4894-ae2f-26e70a843973" Nov 03 23:00:20.800422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.800406 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" Nov 03 23:00:20.800522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.800461 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn" event={"ID":"fc425062-2635-4d50-81fc-56bd9a9dab42","Type":"ContainerDied","Data":"d04cc1f326e732e42aee9a41d6eff9dd3bd487cb347019bf9211c3c7ee105cbe"} Nov 03 23:00:20.802443 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.802423 5272 generic.go:356] "Generic (PLEG): container finished" podID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerID="6c01b9cc41da3fd6e13d40c0c154023a970cc99ca1387bc802cdde55b09e2e4a" exitCode=60 Nov 03 23:00:20.802545 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.802491 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" event={"ID":"5d767797-1338-49e6-b3ad-57fa05f249b5","Type":"ContainerDied","Data":"6c01b9cc41da3fd6e13d40c0c154023a970cc99ca1387bc802cdde55b09e2e4a"} Nov 03 23:00:20.802921 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.802903 5272 scope.go:122] "RemoveContainer" containerID="6c01b9cc41da3fd6e13d40c0c154023a970cc99ca1387bc802cdde55b09e2e4a" Nov 03 23:00:20.803204 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:20.803185 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-dv2fb_odh-ai-gateway-infra(5d767797-1338-49e6-b3ad-57fa05f249b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" Nov 03 23:00:20.804098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.804079 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" Nov 03 23:00:20.804196 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.804101 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8" event={"ID":"126243d7-7eaa-47db-b2da-615c6559e529","Type":"ContainerDied","Data":"1b6b7f1fb71730c75fb9636f2a26a45a1a7a10f4eccf71d3a27672b4a3e1b0f2"} Nov 03 23:00:20.805430 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.805409 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" Nov 03 23:00:20.805543 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.805446 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw" event={"ID":"ed28ee9e-6798-4f37-8020-5325aa4c63cb","Type":"ContainerDied","Data":"47b23ee9310f5acbaeadb1acb1345ada5b9c2b05524779876976956089ea549d"} Nov 03 23:00:20.806976 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.806952 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" event={"ID":"eec2ed53-af75-461b-872c-27f9b7bef027","Type":"ContainerDied","Data":"a81b195dfab0387a89a9286876a8c8efd9d788f12ed117339442e521cdc6f247"} Nov 03 23:00:20.807075 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.806964 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk" Nov 03 23:00:20.815678 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.815662 5272 scope.go:122] "RemoveContainer" containerID="b6f94c7e3db72d0eb80f9f2debafabce536137ada7915e411bc5437c1b4825bb" Nov 03 23:00:20.826137 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.826120 5272 scope.go:122] "RemoveContainer" containerID="3eeca975cbe9c311f98a15c7c107c7a7ed8f4bc1ce282c0d46d2ada7375d4d60" Nov 03 23:00:20.839156 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.839137 5272 scope.go:122] "RemoveContainer" containerID="0aa07e35fff7aaf47835cff83592009490d30aa0e4baec598742029854b0f04a" Nov 03 23:00:20.851137 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.851056 5272 scope.go:122] "RemoveContainer" containerID="b80b6fa5a5d8b9a8ec72f0fe34cc985fed5b55888c15e9ede234bc2910e1c9f3" Nov 03 23:00:20.860590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:20.860570 5272 scope.go:122] "RemoveContainer" containerID="d04570f4e6d625b0fe14d2e8c863954c45d836266eda2d75ff3c0eb9f9cb880e" Nov 03 23:00:21.013775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.013745 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn"] Nov 03 23:00:21.018475 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.018451 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f1190-29895780-kvxrn"] Nov 03 23:00:21.125578 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.125499 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8"] Nov 03 23:00:21.130223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.130192 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-961054-29895780-6d2z8"] Nov 03 23:00:21.339851 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.339819 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw"] Nov 03 23:00:21.343834 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.343780 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-65cac0-29895780-ltzrw"] Nov 03 23:00:21.445683 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.445608 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk"] Nov 03 23:00:21.454145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.454118 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a03fee-29895780-rktpk"] Nov 03 23:00:21.494485 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.494458 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126243d7-7eaa-47db-b2da-615c6559e529" path="/var/lib/kubelet/pods/126243d7-7eaa-47db-b2da-615c6559e529/volumes" Nov 03 23:00:21.494911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.494896 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" path="/var/lib/kubelet/pods/ed28ee9e-6798-4f37-8020-5325aa4c63cb/volumes" Nov 03 23:00:21.495286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.495273 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" path="/var/lib/kubelet/pods/eec2ed53-af75-461b-872c-27f9b7bef027/volumes" Nov 03 23:00:21.496006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.495990 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" path="/var/lib/kubelet/pods/fc425062-2635-4d50-81fc-56bd9a9dab42/volumes" Nov 03 23:00:21.917989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:21.917955 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc"] Nov 03 23:00:22.078472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.078442 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:22.155701 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.155672 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641c211c-d139-4894-ae2f-26e70a843973-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c211c-d139-4894-ae2f-26e70a843973-maas-api-service-ca\") pod \"641c211c-d139-4894-ae2f-26e70a843973\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " Nov 03 23:00:22.155884 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.155739 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641c211c-d139-4894-ae2f-26e70a843973-kube-api-access-svx2d\" (UniqueName: \"kubernetes.io/projected/641c211c-d139-4894-ae2f-26e70a843973-kube-api-access-svx2d\") pod \"641c211c-d139-4894-ae2f-26e70a843973\" (UID: \"641c211c-d139-4894-ae2f-26e70a843973\") " Nov 03 23:00:22.156039 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.156018 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641c211c-d139-4894-ae2f-26e70a843973-maas-api-service-ca" pod "641c211c-d139-4894-ae2f-26e70a843973" (UID: "641c211c-d139-4894-ae2f-26e70a843973"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:22.156167 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.156153 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641c211c-d139-4894-ae2f-26e70a843973-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.157834 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.157812 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641c211c-d139-4894-ae2f-26e70a843973-kube-api-access-svx2d" pod "641c211c-d139-4894-ae2f-26e70a843973" (UID: "641c211c-d139-4894-ae2f-26e70a843973"). InnerVolumeSpecName "kube-api-access-svx2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:22.245593 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.245509 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb"] Nov 03 23:00:22.258161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.258134 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svx2d\" (UniqueName: \"kubernetes.io/projected/641c211c-d139-4894-ae2f-26e70a843973-kube-api-access-svx2d\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.412288 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.412262 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:22.460514 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.460483 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d767797-1338-49e6-b3ad-57fa05f249b5-kube-api-access-48tn8\" (UniqueName: \"kubernetes.io/projected/5d767797-1338-49e6-b3ad-57fa05f249b5-kube-api-access-48tn8\") pod \"5d767797-1338-49e6-b3ad-57fa05f249b5\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " Nov 03 23:00:22.460687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.460671 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d767797-1338-49e6-b3ad-57fa05f249b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d767797-1338-49e6-b3ad-57fa05f249b5-maas-api-service-ca\") pod \"5d767797-1338-49e6-b3ad-57fa05f249b5\" (UID: \"5d767797-1338-49e6-b3ad-57fa05f249b5\") " Nov 03 23:00:22.461053 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.461027 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d767797-1338-49e6-b3ad-57fa05f249b5-maas-api-service-ca" pod "5d767797-1338-49e6-b3ad-57fa05f249b5" (UID: "5d767797-1338-49e6-b3ad-57fa05f249b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:22.462582 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.462562 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d767797-1338-49e6-b3ad-57fa05f249b5-kube-api-access-48tn8" pod "5d767797-1338-49e6-b3ad-57fa05f249b5" (UID: "5d767797-1338-49e6-b3ad-57fa05f249b5"). InnerVolumeSpecName "kube-api-access-48tn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:22.562857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.562794 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48tn8\" (UniqueName: \"kubernetes.io/projected/5d767797-1338-49e6-b3ad-57fa05f249b5-kube-api-access-48tn8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.562857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.562843 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d767797-1338-49e6-b3ad-57fa05f249b5-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.831224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.831135 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" Nov 03 23:00:22.831224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.831168 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc" event={"ID":"641c211c-d139-4894-ae2f-26e70a843973","Type":"ContainerDied","Data":"50931d11b2e3510c939e3dd9221d984c711582f796b47d9a759d872e8c8d5b80"} Nov 03 23:00:22.831461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.831228 5272 scope.go:122] "RemoveContainer" containerID="d0ef38842d46c96923cea6f6dc768f72133f1bb08252dc261d489d2bc54e220d" Nov 03 23:00:22.832850 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.832798 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" Nov 03 23:00:22.832850 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.832834 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb" event={"ID":"5d767797-1338-49e6-b3ad-57fa05f249b5","Type":"ContainerDied","Data":"f2897206e47a1643aedecb12aaa6f998e1749cb8ec2ee98ace543150decb92da"} Nov 03 23:00:22.841605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.841584 5272 scope.go:122] "RemoveContainer" containerID="6c01b9cc41da3fd6e13d40c0c154023a970cc99ca1387bc802cdde55b09e2e4a" Nov 03 23:00:22.989725 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.989689 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc"] Nov 03 23:00:22.993457 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:22.993433 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ee74cb-29895780-gsdlc"] Nov 03 23:00:23.095494 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:23.095436 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb"] Nov 03 23:00:23.100541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:23.100508 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-dv2fb"] Nov 03 23:00:23.494490 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:23.494413 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" path="/var/lib/kubelet/pods/5d767797-1338-49e6-b3ad-57fa05f249b5/volumes" Nov 03 23:00:23.494853 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:23.494836 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641c211c-d139-4894-ae2f-26e70a843973" path="/var/lib/kubelet/pods/641c211c-d139-4894-ae2f-26e70a843973/volumes" Nov 03 23:00:24.502121 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:24.502088 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:00:26.361174 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:26.361137 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q"] Nov 03 23:00:26.466862 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:26.466823 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p"] Nov 03 23:00:27.165737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.165692 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:00:27.333664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.333628 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk"] Nov 03 23:00:27.334518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334492 5272 cpu_manager.go:414] "containerMap: removing container" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerName="cleanup" Nov 03 23:00:27.334518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334514 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334538 5272 cpu_manager.go:414] "containerMap: removing container" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334544 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334552 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334559 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334579 5272 cpu_manager.go:414] "containerMap: removing container" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334585 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334592 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334597 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334604 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334609 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334616 5272 cpu_manager.go:414] "containerMap: removing container" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334620 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334626 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9b745074-da33-4a98-9474-a09fa0ca0704" containerName="payload-pre-processing" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334631 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b745074-da33-4a98-9474-a09fa0ca0704" containerName="payload-pre-processing" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334639 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334646 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334662 5272 cpu_manager.go:414] "containerMap: removing container" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334667 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334673 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334677 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334683 5272 cpu_manager.go:414] "containerMap: removing container" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334687 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334696 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334701 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334708 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334712 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.334712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334721 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334729 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334737 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" containerName="payload-processing" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334742 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" containerName="payload-processing" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334749 5272 cpu_manager.go:414] "containerMap: removing container" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334753 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334763 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334768 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334775 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334779 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334787 5272 cpu_manager.go:414] "containerMap: removing container" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334791 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334813 5272 cpu_manager.go:414] "containerMap: removing container" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334820 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334834 5272 cpu_manager.go:414] "containerMap: removing container" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334843 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334854 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334859 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334866 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334871 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334878 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334882 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334892 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334897 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334905 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.334912 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335021 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335030 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335037 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335043 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335048 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335054 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a650cb-cc75-4d8e-b4bc-5772595e45bb" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335059 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335065 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335074 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335081 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335086 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335092 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335097 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335102 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335107 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b745074-da33-4a98-9474-a09fa0ca0704" containerName="payload-pre-processing" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335113 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335119 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335125 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335130 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335136 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed28ee9e-6798-4f37-8020-5325aa4c63cb" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335145 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d909e24-6172-4e2a-b04e-8a0c7c203a0f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335155 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335160 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b491f0-73d8-4bb9-90ae-1ab6b489b8a9" containerName="payload-processing" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335165 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335171 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335176 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335181 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="600c464b-672f-46bd-8122-0b44bb476a9a" containerName="istio-proxy" Nov 03 23:00:27.335929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.335186 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:00:27.340239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.340221 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.394450 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.392846 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:00:27.394450 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.392926 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:27.394450 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.393015 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:00:27.394450 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.393107 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:00:27.395793 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.395753 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk"] Nov 03 23:00:27.436607 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.436528 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsx9g\" (UniqueName: \"kubernetes.io/projected/3b682ae1-5f26-4d7d-930b-451d818bc172-kube-api-access-jsx9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.436607 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.436569 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.436888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.436722 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.553824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.553518 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsx9g\" (UniqueName: \"kubernetes.io/projected/3b682ae1-5f26-4d7d-930b-451d818bc172-kube-api-access-jsx9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.553824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.553581 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.553824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.553684 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.557877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.554706 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.586349 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.586312 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.665561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.665522 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsx9g\" (UniqueName: \"kubernetes.io/projected/3b682ae1-5f26-4d7d-930b-451d818bc172-kube-api-access-jsx9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:27.958829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:27.957029 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:28.698088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:28.698050 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk"] Nov 03 23:00:29.027956 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:29.027919 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" event={"ID":"3b682ae1-5f26-4d7d-930b-451d818bc172","Type":"ContainerStarted","Data":"9f3dbcf5a76ebff09d4abbc8104ce0a9d832a971d211e106e2da7ff6106d4fb0"} Nov 03 23:00:30.037025 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:30.036990 5272 generic.go:356] "Generic (PLEG): container finished" podID="3b682ae1-5f26-4d7d-930b-451d818bc172" containerID="81950b2487bf3dc6030733753036afc77d2da7db5e300ad3727847c30111a4fa" exitCode=0 Nov 03 23:00:30.037441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:30.037072 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" event={"ID":"3b682ae1-5f26-4d7d-930b-451d818bc172","Type":"ContainerDied","Data":"81950b2487bf3dc6030733753036afc77d2da7db5e300ad3727847c30111a4fa"} Nov 03 23:00:31.244523 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.244491 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:31.370337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.370250 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-openshift-ca\") pod \"3b682ae1-5f26-4d7d-930b-451d818bc172\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " Nov 03 23:00:31.370593 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.370574 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-xks-ca\") pod \"3b682ae1-5f26-4d7d-930b-451d818bc172\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " Nov 03 23:00:31.370981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.370956 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b682ae1-5f26-4d7d-930b-451d818bc172-kube-api-access-jsx9g\" (UniqueName: \"kubernetes.io/projected/3b682ae1-5f26-4d7d-930b-451d818bc172-kube-api-access-jsx9g\") pod \"3b682ae1-5f26-4d7d-930b-451d818bc172\" (UID: \"3b682ae1-5f26-4d7d-930b-451d818bc172\") " Nov 03 23:00:31.371767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.371446 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-openshift-ca" pod "3b682ae1-5f26-4d7d-930b-451d818bc172" (UID: "3b682ae1-5f26-4d7d-930b-451d818bc172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:31.371984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.371957 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" podUID="c875303f-3ae0-46ad-af10-bcfaea6232d5" containerName="payload-processing" containerID="cri-o://9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3" gracePeriod=25 Nov 03 23:00:31.372830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.372432 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:31.375940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.375897 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-xks-ca" pod "3b682ae1-5f26-4d7d-930b-451d818bc172" (UID: "3b682ae1-5f26-4d7d-930b-451d818bc172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:31.376094 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.376074 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b682ae1-5f26-4d7d-930b-451d818bc172-kube-api-access-jsx9g" pod "3b682ae1-5f26-4d7d-930b-451d818bc172" (UID: "3b682ae1-5f26-4d7d-930b-451d818bc172"). InnerVolumeSpecName "kube-api-access-jsx9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:31.469192 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.468928 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" podUID="c60c7dfd-8682-4669-9e6f-6c415fb5f353" containerName="payload-pre-processing" containerID="cri-o://9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d" gracePeriod=25 Nov 03 23:00:31.474319 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.474297 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsx9g\" (UniqueName: \"kubernetes.io/projected/3b682ae1-5f26-4d7d-930b-451d818bc172-kube-api-access-jsx9g\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:31.474441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.474431 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b682ae1-5f26-4d7d-930b-451d818bc172-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:31.789668 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.788852 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 23:00:31.815574 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.815504 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q"] Nov 03 23:00:31.820768 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.820737 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv"] Nov 03 23:00:31.882380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.879849 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl"] Nov 03 23:00:31.882380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.880619 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c875303f-3ae0-46ad-af10-bcfaea6232d5-kube-api-access-rxfnr\" (UniqueName: \"kubernetes.io/projected/c875303f-3ae0-46ad-af10-bcfaea6232d5-kube-api-access-rxfnr\") pod \"c875303f-3ae0-46ad-af10-bcfaea6232d5\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " Nov 03 23:00:31.882380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.880760 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c875303f-3ae0-46ad-af10-bcfaea6232d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875303f-3ae0-46ad-af10-bcfaea6232d5-plugins-config-volume\") pod \"c875303f-3ae0-46ad-af10-bcfaea6232d5\" (UID: \"c875303f-3ae0-46ad-af10-bcfaea6232d5\") " Nov 03 23:00:31.882764 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.882735 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c875303f-3ae0-46ad-af10-bcfaea6232d5-plugins-config-volume" pod "c875303f-3ae0-46ad-af10-bcfaea6232d5" (UID: "c875303f-3ae0-46ad-af10-bcfaea6232d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:31.884018 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.883993 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c875303f-3ae0-46ad-af10-bcfaea6232d5-kube-api-access-rxfnr" pod "c875303f-3ae0-46ad-af10-bcfaea6232d5" (UID: "c875303f-3ae0-46ad-af10-bcfaea6232d5"). InnerVolumeSpecName "kube-api-access-rxfnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:31.904093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.904068 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 23:00:31.982624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.982547 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60c7dfd-8682-4669-9e6f-6c415fb5f353-kube-api-access-ctrjn\" (UniqueName: \"kubernetes.io/projected/c60c7dfd-8682-4669-9e6f-6c415fb5f353-kube-api-access-ctrjn\") pod \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " Nov 03 23:00:31.983035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.983009 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60c7dfd-8682-4669-9e6f-6c415fb5f353-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c7dfd-8682-4669-9e6f-6c415fb5f353-plugins-config-volume\") pod \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\" (UID: \"c60c7dfd-8682-4669-9e6f-6c415fb5f353\") " Nov 03 23:00:31.983911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.983892 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c875303f-3ae0-46ad-af10-bcfaea6232d5-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:31.984039 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.984028 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfnr\" (UniqueName: \"kubernetes.io/projected/c875303f-3ae0-46ad-af10-bcfaea6232d5-kube-api-access-rxfnr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:31.984097 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.983425 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60c7dfd-8682-4669-9e6f-6c415fb5f353-plugins-config-volume" pod "c60c7dfd-8682-4669-9e6f-6c415fb5f353" (UID: "c60c7dfd-8682-4669-9e6f-6c415fb5f353"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:31.984937 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:31.984863 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60c7dfd-8682-4669-9e6f-6c415fb5f353-kube-api-access-ctrjn" pod "c60c7dfd-8682-4669-9e6f-6c415fb5f353" (UID: "c60c7dfd-8682-4669-9e6f-6c415fb5f353"). InnerVolumeSpecName "kube-api-access-ctrjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:32.083385 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.083349 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" Nov 03 23:00:32.083555 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.083389 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk" event={"ID":"3b682ae1-5f26-4d7d-930b-451d818bc172","Type":"ContainerDied","Data":"9f3dbcf5a76ebff09d4abbc8104ce0a9d832a971d211e106e2da7ff6106d4fb0"} Nov 03 23:00:32.083555 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.083464 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3dbcf5a76ebff09d4abbc8104ce0a9d832a971d211e106e2da7ff6106d4fb0" Nov 03 23:00:32.084767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.084743 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctrjn\" (UniqueName: \"kubernetes.io/projected/c60c7dfd-8682-4669-9e6f-6c415fb5f353-kube-api-access-ctrjn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:32.084767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.084765 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60c7dfd-8682-4669-9e6f-6c415fb5f353-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:32.086019 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.085996 5272 generic.go:356] "Generic (PLEG): container finished" podID="c60c7dfd-8682-4669-9e6f-6c415fb5f353" containerID="9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d" exitCode=0 Nov 03 23:00:32.086105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.086050 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" Nov 03 23:00:32.086105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.086061 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" event={"ID":"c60c7dfd-8682-4669-9e6f-6c415fb5f353","Type":"ContainerDied","Data":"9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d"} Nov 03 23:00:32.086224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.086119 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p" event={"ID":"c60c7dfd-8682-4669-9e6f-6c415fb5f353","Type":"ContainerDied","Data":"acbf6361d9572e3f4da5f39202b3800204621bc1e64b0e1ae41844f5db2e483f"} Nov 03 23:00:32.086224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.086150 5272 scope.go:122] "RemoveContainer" containerID="9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d" Nov 03 23:00:32.092784 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.092763 5272 generic.go:356] "Generic (PLEG): container finished" podID="c875303f-3ae0-46ad-af10-bcfaea6232d5" containerID="9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3" exitCode=0 Nov 03 23:00:32.092895 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.092823 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" event={"ID":"c875303f-3ae0-46ad-af10-bcfaea6232d5","Type":"ContainerDied","Data":"9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3"} Nov 03 23:00:32.092895 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.092846 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" Nov 03 23:00:32.092895 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.092856 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q" event={"ID":"c875303f-3ae0-46ad-af10-bcfaea6232d5","Type":"ContainerDied","Data":"c5069577a867076c747bc579e25bbe29d68410c61f487487c7d55531a85f1075"} Nov 03 23:00:32.099044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.099021 5272 scope.go:122] "RemoveContainer" containerID="9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d" Nov 03 23:00:32.099321 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:32.099300 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d\": container with ID starting with 9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d not found: ID does not exist" containerID="9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d" Nov 03 23:00:32.099401 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.099327 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d"} err="failed to get container status \"9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d\": rpc error: code = NotFound desc = could not find container \"9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d\": container with ID starting with 9ba0a08d7557697b26c4feb037549af001c3f38a5ee9b9b8df467cf84dd28c4d not found: ID does not exist" Nov 03 23:00:32.099401 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.099344 5272 scope.go:122] "RemoveContainer" containerID="9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3" Nov 03 23:00:32.111577 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.111549 5272 scope.go:122] "RemoveContainer" containerID="9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3" Nov 03 23:00:32.111879 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:32.111860 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3\": container with ID starting with 9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3 not found: ID does not exist" containerID="9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3" Nov 03 23:00:32.111932 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.111885 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3"} err="failed to get container status \"9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3\": rpc error: code = NotFound desc = could not find container \"9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3\": container with ID starting with 9c1c3813515194fe94c58b8355b6f0f018acaeede1505b488069aa7afd5862f3 not found: ID does not exist" Nov 03 23:00:32.498093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.498060 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p"] Nov 03 23:00:32.502486 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.502452 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-65cac0-85d8c6fb67-ln66p"] Nov 03 23:00:32.676138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.676092 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q"] Nov 03 23:00:32.680641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:32.680598 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-65cac0-7bbdd5876-c484q"] Nov 03 23:00:33.495486 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:33.495453 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60c7dfd-8682-4669-9e6f-6c415fb5f353" path="/var/lib/kubelet/pods/c60c7dfd-8682-4669-9e6f-6c415fb5f353/volumes" Nov 03 23:00:33.495968 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:33.495951 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c875303f-3ae0-46ad-af10-bcfaea6232d5" path="/var/lib/kubelet/pods/c875303f-3ae0-46ad-af10-bcfaea6232d5/volumes" Nov 03 23:00:37.147208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:37.147179 5272 generic.go:356] "Generic (PLEG): container finished" podID="f9060091-788b-47ed-8934-18fc8c9bc462" containerID="cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c" exitCode=0 Nov 03 23:00:37.147668 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:37.147245 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" event={"ID":"f9060091-788b-47ed-8934-18fc8c9bc462","Type":"ContainerDied","Data":"cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c"} Nov 03 23:00:37.147773 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:37.147755 5272 scope.go:122] "RemoveContainer" containerID="cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c" Nov 03 23:00:37.149697 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:37.149659 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:00:37.149786 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:37.149765 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:00:37.149858 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:37.149841 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:00:38.173949 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:38.173909 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" event={"ID":"f9060091-788b-47ed-8934-18fc8c9bc462","Type":"ContainerStarted","Data":"2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce"} Nov 03 23:00:38.770166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:38.770128 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh"] Nov 03 23:00:39.099707 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:39.099671 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 23:00:39.099913 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:39.099889 5272 patch_prober.go:29] interesting pod/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 23:00:39.099978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:39.099953 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 23:00:40.099476 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:40.099443 5272 patch_prober.go:29] interesting pod/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 23:00:40.099907 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:40.099503 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 23:00:41.099900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:41.099864 5272 patch_prober.go:29] interesting pod/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 23:00:41.100315 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:41.099922 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 23:00:42.099089 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:42.099050 5272 patch_prober.go:29] interesting pod/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 23:00:42.099334 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:42.099111 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 23:00:43.019510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:43.019479 5272 scope.go:122] "RemoveContainer" containerID="695a8d18361b511b2585fd2fee4157455d60d7796cc73c20b6ea029eadefd7d6" Nov 03 23:00:43.029627 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:43.029607 5272 scope.go:122] "RemoveContainer" containerID="69bccd22ebcfbec10f285a677b8cc5e6d21eccb9cc4ee191b73f7a0d19806294" Nov 03 23:00:43.127836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:43.123294 5272 patch_prober.go:29] interesting pod/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 03 23:00:43.127836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:43.123363 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 03 23:00:43.771030 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:43.770991 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" podUID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" containerName="payload-processing" containerID="cri-o://da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d" gracePeriod=25 Nov 03 23:00:44.103460 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:44.103427 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 23:00:44.103987 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:44.103772 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 23:00:44.104425 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:44.104404 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 23:00:46.869102 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:46.869037 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" containerID="cri-o://07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0" gracePeriod=45 Nov 03 23:00:46.896810 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:46.896751 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" containerID="cri-o://d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb" gracePeriod=45 Nov 03 23:00:46.906399 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:46.906368 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": dial tcp 10.133.0.84:8000: connect: connection refused" Nov 03 23:00:46.907079 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:46.907048 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" containerID="cri-o://e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a" gracePeriod=45 Nov 03 23:00:47.119650 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.119588 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused" Nov 03 23:00:47.200267 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.200176 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 23:00:47.206903 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.206874 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 23:00:47.211735 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.211715 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 23:00:47.266383 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.266351 5272 generic.go:356] "Generic (PLEG): container finished" podID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerID="07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0" exitCode=0 Nov 03 23:00:47.266564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.266394 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" Nov 03 23:00:47.266564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.266394 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" event={"ID":"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55","Type":"ContainerDied","Data":"07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0"} Nov 03 23:00:47.266564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.266507 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q" event={"ID":"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55","Type":"ContainerDied","Data":"cdb936f620b7ad139c5849f18cd4e9ee57babe356441baa562f9310c1f0354e1"} Nov 03 23:00:47.266564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.266529 5272 scope.go:122] "RemoveContainer" containerID="07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0" Nov 03 23:00:47.267946 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.267922 5272 generic.go:356] "Generic (PLEG): container finished" podID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerID="e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a" exitCode=0 Nov 03 23:00:47.268099 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.267942 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" event={"ID":"1a2d866d-b309-4640-98f7-a8e9e8f94a25","Type":"ContainerDied","Data":"e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a"} Nov 03 23:00:47.268099 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.267978 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" event={"ID":"1a2d866d-b309-4640-98f7-a8e9e8f94a25","Type":"ContainerDied","Data":"5f38a23f8c92e8e9e4aaaffdc02731f4ce1f0bb7892de80d10fa9f74d5b9b7b0"} Nov 03 23:00:47.268099 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.268023 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl" Nov 03 23:00:47.269895 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.269869 5272 generic.go:356] "Generic (PLEG): container finished" podID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerID="d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb" exitCode=0 Nov 03 23:00:47.270028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.269916 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" Nov 03 23:00:47.270028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.269947 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" event={"ID":"b266a56d-c69e-4051-94c8-6037b6b5d21e","Type":"ContainerDied","Data":"d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb"} Nov 03 23:00:47.270028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.269981 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv" event={"ID":"b266a56d-c69e-4051-94c8-6037b6b5d21e","Type":"ContainerDied","Data":"a4be060de5d3cf76f49a463cddfc69e0f6cd083f4d0445415476d564b93907e0"} Nov 03 23:00:47.276792 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.276775 5272 scope.go:122] "RemoveContainer" containerID="07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0" Nov 03 23:00:47.277104 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:47.277083 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0\": container with ID starting with 07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0 not found: ID does not exist" containerID="07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0" Nov 03 23:00:47.277200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.277116 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0"} err="failed to get container status \"07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0\": rpc error: code = NotFound desc = could not find container \"07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0\": container with ID starting with 07330db68d21164ae881ec8a8db4d2645c7b994e99550a6ccda77fc70e32bae0 not found: ID does not exist" Nov 03 23:00:47.277200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.277137 5272 scope.go:122] "RemoveContainer" containerID="e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a" Nov 03 23:00:47.286300 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.286285 5272 scope.go:122] "RemoveContainer" containerID="e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a" Nov 03 23:00:47.286549 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:47.286530 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a\": container with ID starting with e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a not found: ID does not exist" containerID="e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a" Nov 03 23:00:47.286614 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.286554 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a"} err="failed to get container status \"e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a\": rpc error: code = NotFound desc = could not find container \"e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a\": container with ID starting with e167c37f2db5b3e1da59845b37fa50e4d167eaec7fd4ccd29c3f4fa4d41e550a not found: ID does not exist" Nov 03 23:00:47.286614 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.286571 5272 scope.go:122] "RemoveContainer" containerID="d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb" Nov 03 23:00:47.294960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.294941 5272 scope.go:122] "RemoveContainer" containerID="d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb" Nov 03 23:00:47.295211 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:47.295194 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb\": container with ID starting with d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb not found: ID does not exist" containerID="d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb" Nov 03 23:00:47.295279 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.295216 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb"} err="failed to get container status \"d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb\": rpc error: code = NotFound desc = could not find container \"d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb\": container with ID starting with d262dce156c856742d401283140b7a921aeb7b89b6fa2ef048c8b1863a6002fb not found: ID does not exist" Nov 03 23:00:47.332103 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332076 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-home\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-home\") pod \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " Nov 03 23:00:47.332239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332121 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tls-certs\") pod \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " Nov 03 23:00:47.332239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332161 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2d866d-b309-4640-98f7-a8e9e8f94a25-kube-api-access-cnkb2\" (UniqueName: \"kubernetes.io/projected/1a2d866d-b309-4640-98f7-a8e9e8f94a25-kube-api-access-cnkb2\") pod \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " Nov 03 23:00:47.332239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332207 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-tmp-dir\") pod \"b266a56d-c69e-4051-94c8-6037b6b5d21e\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " Nov 03 23:00:47.332239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332235 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tmp-dir\") pod \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " Nov 03 23:00:47.332479 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332272 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-dshm\") pod \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " Nov 03 23:00:47.332479 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332321 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-model-cache\") pod \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " Nov 03 23:00:47.332479 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332352 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tls-certs\") pod \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " Nov 03 23:00:47.332479 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332374 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-home" pod "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" (UID: "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.332479 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332381 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b266a56d-c69e-4051-94c8-6037b6b5d21e-kube-api-access-t2qkf\" (UniqueName: \"kubernetes.io/projected/b266a56d-c69e-4051-94c8-6037b6b5d21e-kube-api-access-t2qkf\") pod \"b266a56d-c69e-4051-94c8-6037b6b5d21e\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " Nov 03 23:00:47.332479 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332454 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-dshm\") pod \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332545 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-dshm\") pod \"b266a56d-c69e-4051-94c8-6037b6b5d21e\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332674 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-kube-api-access-4rgts\" (UniqueName: \"kubernetes.io/projected/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-kube-api-access-4rgts\") pod \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\" (UID: \"4cb4b4c2-02ad-468e-ba1a-7625d94a2f55\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332713 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-model-cache\") pod \"b266a56d-c69e-4051-94c8-6037b6b5d21e\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332757 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-home\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-home\") pod \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332832 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b266a56d-c69e-4051-94c8-6037b6b5d21e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b266a56d-c69e-4051-94c8-6037b6b5d21e-tls-certs\") pod \"b266a56d-c69e-4051-94c8-6037b6b5d21e\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332914 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-home\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-home\") pod \"b266a56d-c69e-4051-94c8-6037b6b5d21e\" (UID: \"b266a56d-c69e-4051-94c8-6037b6b5d21e\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.332950 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tmp-dir\") pod \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " Nov 03 23:00:47.333142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.333016 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-model-cache\") pod \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\" (UID: \"1a2d866d-b309-4640-98f7-a8e9e8f94a25\") " Nov 03 23:00:47.333740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.333713 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.334231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.334031 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-model-cache" pod "1a2d866d-b309-4640-98f7-a8e9e8f94a25" (UID: "1a2d866d-b309-4640-98f7-a8e9e8f94a25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.334231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.334213 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-home" pod "1a2d866d-b309-4640-98f7-a8e9e8f94a25" (UID: "1a2d866d-b309-4640-98f7-a8e9e8f94a25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.334968 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.334581 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-model-cache" pod "b266a56d-c69e-4051-94c8-6037b6b5d21e" (UID: "b266a56d-c69e-4051-94c8-6037b6b5d21e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.334968 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.334840 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-home" pod "b266a56d-c69e-4051-94c8-6037b6b5d21e" (UID: "b266a56d-c69e-4051-94c8-6037b6b5d21e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.334968 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.334918 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tmp-dir" pod "1a2d866d-b309-4640-98f7-a8e9e8f94a25" (UID: "1a2d866d-b309-4640-98f7-a8e9e8f94a25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.335190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.334995 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-tmp-dir" pod "b266a56d-c69e-4051-94c8-6037b6b5d21e" (UID: "b266a56d-c69e-4051-94c8-6037b6b5d21e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.335333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.335258 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tls-certs" pod "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" (UID: "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:47.335333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.335306 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-model-cache" pod "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" (UID: "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.335506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.335478 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tmp-dir" pod "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" (UID: "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.335709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.335685 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b266a56d-c69e-4051-94c8-6037b6b5d21e-kube-api-access-t2qkf" pod "b266a56d-c69e-4051-94c8-6037b6b5d21e" (UID: "b266a56d-c69e-4051-94c8-6037b6b5d21e"). InnerVolumeSpecName "kube-api-access-t2qkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:47.336525 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.336499 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tls-certs" pod "1a2d866d-b309-4640-98f7-a8e9e8f94a25" (UID: "1a2d866d-b309-4640-98f7-a8e9e8f94a25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:47.337112 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.337030 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b266a56d-c69e-4051-94c8-6037b6b5d21e-tls-certs" pod "b266a56d-c69e-4051-94c8-6037b6b5d21e" (UID: "b266a56d-c69e-4051-94c8-6037b6b5d21e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:47.337217 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.337145 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-dshm" pod "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" (UID: "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.337407 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.337379 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-kube-api-access-4rgts" pod "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" (UID: "4cb4b4c2-02ad-468e-ba1a-7625d94a2f55"). InnerVolumeSpecName "kube-api-access-4rgts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:47.338088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.338063 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-dshm" pod "1a2d866d-b309-4640-98f7-a8e9e8f94a25" (UID: "1a2d866d-b309-4640-98f7-a8e9e8f94a25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.338373 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.338356 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-dshm" pod "b266a56d-c69e-4051-94c8-6037b6b5d21e" (UID: "b266a56d-c69e-4051-94c8-6037b6b5d21e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:47.338755 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.338739 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2d866d-b309-4640-98f7-a8e9e8f94a25-kube-api-access-cnkb2" pod "1a2d866d-b309-4640-98f7-a8e9e8f94a25" (UID: "1a2d866d-b309-4640-98f7-a8e9e8f94a25"). InnerVolumeSpecName "kube-api-access-cnkb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:47.434737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434664 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.434737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434694 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rgts\" (UniqueName: \"kubernetes.io/projected/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-kube-api-access-4rgts\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.434737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434705 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.434737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434715 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.434737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434727 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b266a56d-c69e-4051-94c8-6037b6b5d21e-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.434737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434738 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434747 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434755 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434764 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434772 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkb2\" (UniqueName: \"kubernetes.io/projected/1a2d866d-b309-4640-98f7-a8e9e8f94a25-kube-api-access-cnkb2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434781 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b266a56d-c69e-4051-94c8-6037b6b5d21e-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434789 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434797 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2d866d-b309-4640-98f7-a8e9e8f94a25-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434819 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434829 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2d866d-b309-4640-98f7-a8e9e8f94a25-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434839 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qkf\" (UniqueName: \"kubernetes.io/projected/b266a56d-c69e-4051-94c8-6037b6b5d21e-kube-api-access-t2qkf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.435087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.434848 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:47.684359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.684328 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv"] Nov 03 23:00:47.690563 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.690497 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-961054/premium-w3-961054-kserve-d8c56d8cd-t2hsv"] Nov 03 23:00:47.790141 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.790115 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl"] Nov 03 23:00:47.795028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.795005 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-961054/simulator-w3-961054-kserve-84586b6dd7-wkssl"] Nov 03 23:00:47.896544 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.896515 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q"] Nov 03 23:00:47.900531 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:47.900502 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-961054/distinct-w3-961054-kserve-65d7477cd5-dcp2q"] Nov 03 23:00:49.494503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:49.494473 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" path="/var/lib/kubelet/pods/1a2d866d-b309-4640-98f7-a8e9e8f94a25/volumes" Nov 03 23:00:49.494947 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:49.494926 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" path="/var/lib/kubelet/pods/4cb4b4c2-02ad-468e-ba1a-7625d94a2f55/volumes" Nov 03 23:00:49.495337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:49.495322 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" path="/var/lib/kubelet/pods/b266a56d-c69e-4051-94c8-6037b6b5d21e/volumes" Nov 03 23:00:49.896569 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:49.896534 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" podUID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:00:54.501052 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:00:54.501014 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:00:59.896472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:00:59.896431 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" podUID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:01:01.223732 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:01.223699 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4"] Nov 03 23:01:01.270383 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:01.270351 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6296b-kflt4"] Nov 03 23:01:01.493380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:01.493290 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0449308a-4743-4b1e-94bd-fa1cf13cdca7" path="/var/lib/kubelet/pods/0449308a-4743-4b1e-94bd-fa1cf13cdca7/volumes" Nov 03 23:01:04.026230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.026209 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 23:01:04.173886 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.173854 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-plugins-config-volume\") pod \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " Nov 03 23:01:04.174053 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.173976 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-kube-api-access-k52wn\" (UniqueName: \"kubernetes.io/projected/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-kube-api-access-k52wn\") pod \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\" (UID: \"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd\") " Nov 03 23:01:04.174274 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.174248 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-plugins-config-volume" pod "744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" (UID: "744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:04.174477 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.174464 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:04.176003 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.175976 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-kube-api-access-k52wn" pod "744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" (UID: "744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd"). InnerVolumeSpecName "kube-api-access-k52wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:04.275374 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.275350 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52wn\" (UniqueName: \"kubernetes.io/projected/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd-kube-api-access-k52wn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:04.389557 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.389523 5272 generic.go:356] "Generic (PLEG): container finished" podID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" containerID="da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d" exitCode=0 Nov 03 23:01:04.389742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.389567 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" Nov 03 23:01:04.389742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.389609 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" event={"ID":"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd","Type":"ContainerDied","Data":"da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d"} Nov 03 23:01:04.389742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.389667 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh" event={"ID":"744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd","Type":"ContainerDied","Data":"22215cd852e31f874f8ec58b5338a8c7f9a097887e0cb6e254240bf7275cc71c"} Nov 03 23:01:04.389742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.389695 5272 scope.go:122] "RemoveContainer" containerID="da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d" Nov 03 23:01:04.399474 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.399458 5272 scope.go:122] "RemoveContainer" containerID="da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d" Nov 03 23:01:04.399753 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:01:04.399734 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d\": container with ID starting with da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d not found: ID does not exist" containerID="da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d" Nov 03 23:01:04.399854 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.399761 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d"} err="failed to get container status \"da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d\": rpc error: code = NotFound desc = could not find container \"da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d\": container with ID starting with da16482b131539bca59945949daca4a241f66e68ef9808354e7ba5137442cf9d not found: ID does not exist" Nov 03 23:01:04.544709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.544671 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh"] Nov 03 23:01:04.549064 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:04.549040 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-961054-66867965c8-8zxjh"] Nov 03 23:01:05.493303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:05.493269 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" path="/var/lib/kubelet/pods/744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd/volumes" Nov 03 23:01:09.473223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473188 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8"] Nov 03 23:01:09.473928 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473908 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" Nov 03 23:01:09.473928 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473929 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473945 5272 cpu_manager.go:414] "containerMap: removing container" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473953 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473971 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473977 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473985 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.473991 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32b4f71-9f49-4f92-b766-88ce76449123" containerName="istio-proxy" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474007 5272 cpu_manager.go:414] "containerMap: removing container" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474011 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126243d7-7eaa-47db-b2da-615c6559e529" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474021 5272 cpu_manager.go:414] "containerMap: removing container" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474026 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474034 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c60c7dfd-8682-4669-9e6f-6c415fb5f353" containerName="payload-pre-processing" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474039 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60c7dfd-8682-4669-9e6f-6c415fb5f353" containerName="payload-pre-processing" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474051 5272 cpu_manager.go:414] "containerMap: removing container" podUID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" containerName="payload-processing" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474056 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" containerName="payload-processing" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474066 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c875303f-3ae0-46ad-af10-bcfaea6232d5" containerName="payload-processing" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474071 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c875303f-3ae0-46ad-af10-bcfaea6232d5" containerName="payload-processing" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474083 5272 cpu_manager.go:414] "containerMap: removing container" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474087 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerName="cleanup" Nov 03 23:01:09.474088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474097 5272 cpu_manager.go:414] "containerMap: removing container" podUID="3b682ae1-5f26-4d7d-930b-451d818bc172" containerName="revoke-keys" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474102 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b682ae1-5f26-4d7d-930b-451d818bc172" containerName="revoke-keys" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474111 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474116 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474123 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474127 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474139 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474143 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474149 5272 cpu_manager.go:414] "containerMap: removing container" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474155 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474246 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="641c211c-d139-4894-ae2f-26e70a843973" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474255 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b266a56d-c69e-4051-94c8-6037b6b5d21e" containerName="main" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474262 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6781d3-1e50-4b96-8874-2dc962a89a4f" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474271 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60c7dfd-8682-4669-9e6f-6c415fb5f353" containerName="payload-pre-processing" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474277 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="744ce8f2-fa9c-4eb7-a877-3fc09a15e1cd" containerName="payload-processing" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474284 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d767797-1338-49e6-b3ad-57fa05f249b5" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474289 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c875303f-3ae0-46ad-af10-bcfaea6232d5" containerName="payload-processing" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474296 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="38737a60-c402-4a2a-b52a-d765dcc36e1c" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474302 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="09966415-3039-4b0f-8ab5-47f1d1e8cdf0" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474308 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b682ae1-5f26-4d7d-930b-451d818bc172" containerName="revoke-keys" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474313 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2d866d-b309-4640-98f7-a8e9e8f94a25" containerName="main" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474319 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec2ed53-af75-461b-872c-27f9b7bef027" containerName="cleanup" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474326 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb4b4c2-02ad-468e-ba1a-7625d94a2f55" containerName="main" Nov 03 23:01:09.474906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.474333 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc425062-2635-4d50-81fc-56bd9a9dab42" containerName="cleanup" Nov 03 23:01:09.479217 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.479198 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.530177 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.530128 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0bf80c-openshift-default-dockercfg-lcrrz\"" Nov 03 23:01:09.636189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.636151 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8"] Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.662717 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-envoy\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.662791 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-podinfo\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.662864 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.662902 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cwh\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-kube-api-access-s7cwh\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.663021 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-credential-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.663068 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-certs\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.663113 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-data\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.663266 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istiod-ca-cert\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.669832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.663296 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-token\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765034 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.764956 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-credential-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765034 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765001 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-certs\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765034 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765033 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-data\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765310 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765116 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istiod-ca-cert\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765310 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765143 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-token\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765310 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765209 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-envoy\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765310 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765245 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-podinfo\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765310 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765286 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765588 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765322 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cwh\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-kube-api-access-s7cwh\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765588 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765464 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-credential-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765588 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765483 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-certs\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765757 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765646 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-data\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.765864 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.765757 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-socket\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.766039 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.766016 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istiod-ca-cert\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.767749 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.767719 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-envoy\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.767936 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.767918 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-podinfo\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.822892 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.822860 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-token\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:09.823041 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:09.823024 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cwh\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-kube-api-access-s7cwh\") pod \"e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:10.093017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:10.092987 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:10.586469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:10.574172 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8"] Nov 03 23:01:10.640716 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:10.634995 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:10.641071 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:10.641031 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:10.641244 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:10.641223 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:01:11.476085 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:11.476049 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" event={"ID":"10f3ea73-2aeb-40ed-990a-31a494afd1ed","Type":"ContainerStarted","Data":"0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972"} Nov 03 23:01:11.476306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:11.476106 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" event={"ID":"10f3ea73-2aeb-40ed-990a-31a494afd1ed","Type":"ContainerStarted","Data":"5be775a3e5871f50b499455840a28001db92ba317f2836eda75c59524d717826"} Nov 03 23:01:11.591723 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:11.591670 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podStartSLOduration=2.591656652 podStartE2EDuration="2.591656652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:11.589168485 +0000 UTC m=+2252.211476794" watchObservedRunningTime="2026-11-03 23:01:11.591656652 +0000 UTC m=+2252.213964958" Nov 03 23:01:12.093854 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:12.093820 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:12.099585 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:12.099559 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:12.482547 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:12.482461 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:12.483380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:12.483361 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:01:18.703013 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.702971 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47"] Nov 03 23:01:18.709820 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.709773 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:18.768164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.768126 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47"] Nov 03 23:01:18.768828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.768790 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0bf80c\"" Nov 03 23:01:18.769174 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.769155 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0bf80c-dockercfg-vtfd7\"" Nov 03 23:01:18.834825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.833940 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzlt\" (UniqueName: \"kubernetes.io/projected/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-kube-api-access-7qzlt\") pod \"payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:18.834825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.833997 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:18.859477 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.859443 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp"] Nov 03 23:01:18.863830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.863790 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:18.920601 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.920561 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp"] Nov 03 23:01:18.935659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.935619 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzlt\" (UniqueName: \"kubernetes.io/projected/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-kube-api-access-7qzlt\") pod \"payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:18.935874 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.935690 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:18.935874 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.935733 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:18.935874 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.935825 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklkm\" (UniqueName: \"kubernetes.io/projected/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-kube-api-access-jklkm\") pod \"payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:18.936842 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.936791 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:18.995764 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:18.995689 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzlt\" (UniqueName: \"kubernetes.io/projected/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-kube-api-access-7qzlt\") pod \"payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:19.035638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.035600 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:19.037181 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.037144 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:19.037321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.037227 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklkm\" (UniqueName: \"kubernetes.io/projected/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-kube-api-access-jklkm\") pod \"payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:19.038060 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.038029 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:19.125671 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.125601 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklkm\" (UniqueName: \"kubernetes.io/projected/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-kube-api-access-jklkm\") pod \"payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:19.182349 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.182053 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:19.319067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.318895 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:01:19.553957 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.553918 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47"] Nov 03 23:01:19.560212 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:19.560184 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2ed7ac_8c96_401f_aeb2_7eab48a2e1dc.slice/crio-ab9457529c557c2b867b09fb352d2ae21c7b07f74e08ae7f8fd39e53e51f0694 WatchSource:0}: Error finding container ab9457529c557c2b867b09fb352d2ae21c7b07f74e08ae7f8fd39e53e51f0694: Status 404 returned error can't find the container with id ab9457529c557c2b867b09fb352d2ae21c7b07f74e08ae7f8fd39e53e51f0694 Nov 03 23:01:19.694824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:19.690858 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp"] Nov 03 23:01:20.598829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.598439 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" event={"ID":"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc","Type":"ContainerStarted","Data":"267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d"} Nov 03 23:01:20.598829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.598506 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" event={"ID":"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc","Type":"ContainerStarted","Data":"ab9457529c557c2b867b09fb352d2ae21c7b07f74e08ae7f8fd39e53e51f0694"} Nov 03 23:01:20.606120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.599541 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:20.606120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.602188 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" event={"ID":"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5","Type":"ContainerStarted","Data":"4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c"} Nov 03 23:01:20.606120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.602223 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" event={"ID":"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5","Type":"ContainerStarted","Data":"c4c60a122df57e826a691a07a7360ec16e9b7ac7fe49670f98cbe35751ef66c8"} Nov 03 23:01:20.606120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.602752 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:20.708913 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.708852 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" podStartSLOduration=2.708835332 podStartE2EDuration="2.708835332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:20.708259029 +0000 UTC m=+2261.330567337" watchObservedRunningTime="2026-11-03 23:01:20.708835332 +0000 UTC m=+2261.331143638" Nov 03 23:01:20.821338 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:20.821270 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" podStartSLOduration=2.821253934 podStartE2EDuration="2.821253934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:20.81688333 +0000 UTC m=+2261.439191641" watchObservedRunningTime="2026-11-03 23:01:20.821253934 +0000 UTC m=+2261.443562241" Nov 03 23:01:22.281827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.281733 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4"] Nov 03 23:01:22.313824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.309042 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.333566 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.333376 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4"] Nov 03 23:01:22.337012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.336984 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq"] Nov 03 23:01:22.341843 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.341820 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.440046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.439588 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.440046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.439661 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.440046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.439734 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mtd\" (UniqueName: \"kubernetes.io/projected/06db652e-54ce-4405-b7af-fdcf65b8bf0b-kube-api-access-v6mtd\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.440046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.439788 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.440046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.439873 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9g4\" (UniqueName: \"kubernetes.io/projected/913fcaca-efec-4e75-a731-5aade94d4831-kube-api-access-4f9g4\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.440046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.439971 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.515828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.515531 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:01:22.516064 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.515961 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:01:22.516147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.516064 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:22.516213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.516196 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:22.520541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.520509 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq"] Nov 03 23:01:22.540851 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.540594 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.540851 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.540628 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.541117 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.541092 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mtd\" (UniqueName: \"kubernetes.io/projected/06db652e-54ce-4405-b7af-fdcf65b8bf0b-kube-api-access-v6mtd\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.541278 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.541259 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.541429 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.541412 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9g4\" (UniqueName: \"kubernetes.io/projected/913fcaca-efec-4e75-a731-5aade94d4831-kube-api-access-4f9g4\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.541583 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.541335 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.541674 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.541608 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.541964 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.541942 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.558832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.554270 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.581900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.581873 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.611689 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.611664 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9g4\" (UniqueName: \"kubernetes.io/projected/913fcaca-efec-4e75-a731-5aade94d4831-kube-api-access-4f9g4\") pod \"maas-api-revoke-sub-56b8d6a7-bscmq\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:22.615475 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.615439 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mtd\" (UniqueName: \"kubernetes.io/projected/06db652e-54ce-4405-b7af-fdcf65b8bf0b-kube-api-access-v6mtd\") pod \"maas-api-revoke-sub-ebd1d6fc-9qlh4\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.638711 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.638678 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:22.825194 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:22.825097 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:23.226827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.226177 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4"] Nov 03 23:01:23.232286 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:23.232252 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06db652e_54ce_4405_b7af_fdcf65b8bf0b.slice/crio-9b420ff1d5749f2cdf5fcd29df318af7b35dfceda0917a2ac94fbf99e45f5c23 WatchSource:0}: Error finding container 9b420ff1d5749f2cdf5fcd29df318af7b35dfceda0917a2ac94fbf99e45f5c23: Status 404 returned error can't find the container with id 9b420ff1d5749f2cdf5fcd29df318af7b35dfceda0917a2ac94fbf99e45f5c23 Nov 03 23:01:23.413613 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.413574 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq"] Nov 03 23:01:23.420408 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:23.420379 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913fcaca_efec_4e75_a731_5aade94d4831.slice/crio-68c53a166ca81f509704ffbc1825e65017178fe6d3f61cd976f01e3ff95a311c WatchSource:0}: Error finding container 68c53a166ca81f509704ffbc1825e65017178fe6d3f61cd976f01e3ff95a311c: Status 404 returned error can't find the container with id 68c53a166ca81f509704ffbc1825e65017178fe6d3f61cd976f01e3ff95a311c Nov 03 23:01:23.639332 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.639259 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" event={"ID":"06db652e-54ce-4405-b7af-fdcf65b8bf0b","Type":"ContainerStarted","Data":"89178ca87a5c3bee115f56205134bb817613f85e82d07bf7af7adf8e338676e6"} Nov 03 23:01:23.639332 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.639317 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" event={"ID":"06db652e-54ce-4405-b7af-fdcf65b8bf0b","Type":"ContainerStarted","Data":"9b420ff1d5749f2cdf5fcd29df318af7b35dfceda0917a2ac94fbf99e45f5c23"} Nov 03 23:01:23.640670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.640643 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" event={"ID":"913fcaca-efec-4e75-a731-5aade94d4831","Type":"ContainerStarted","Data":"68c53a166ca81f509704ffbc1825e65017178fe6d3f61cd976f01e3ff95a311c"} Nov 03 23:01:23.985368 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.985224 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" podStartSLOduration=1.9852076090000002 podStartE2EDuration="1.985207609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:23.760466394 +0000 UTC m=+2264.382774704" watchObservedRunningTime="2026-11-03 23:01:23.985207609 +0000 UTC m=+2264.607515918" Nov 03 23:01:23.988364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.988338 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4"] Nov 03 23:01:23.993013 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:23.992995 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.042701 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.042673 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-656ee6\"" Nov 03 23:01:24.042946 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.042873 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-656ee6-dockercfg-d57tt\"" Nov 03 23:01:24.047230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.047204 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4"] Nov 03 23:01:24.154832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.154742 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz"] Nov 03 23:01:24.172944 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.172902 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fbef3b-965d-4a61-bebc-63f605f0b47f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.173250 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.173226 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjf7\" (UniqueName: \"kubernetes.io/projected/14fbef3b-965d-4a61-bebc-63f605f0b47f-kube-api-access-4kjf7\") pod \"payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.207784 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.205861 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz"] Nov 03 23:01:24.207784 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.205998 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.274966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.274878 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fbef3b-965d-4a61-bebc-63f605f0b47f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.274966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.274919 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95901769-38a7-4687-ae42-79dc16ab6c6a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.274966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.274949 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtzg\" (UniqueName: \"kubernetes.io/projected/95901769-38a7-4687-ae42-79dc16ab6c6a-kube-api-access-8qtzg\") pod \"payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.275276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.275022 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjf7\" (UniqueName: \"kubernetes.io/projected/14fbef3b-965d-4a61-bebc-63f605f0b47f-kube-api-access-4kjf7\") pod \"payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.275768 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.275746 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fbef3b-965d-4a61-bebc-63f605f0b47f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.332068 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.332041 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjf7\" (UniqueName: \"kubernetes.io/projected/14fbef3b-965d-4a61-bebc-63f605f0b47f-kube-api-access-4kjf7\") pod \"payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.376475 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.376446 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95901769-38a7-4687-ae42-79dc16ab6c6a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.376610 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.376490 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtzg\" (UniqueName: \"kubernetes.io/projected/95901769-38a7-4687-ae42-79dc16ab6c6a-kube-api-access-8qtzg\") pod \"payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.377256 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.377225 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95901769-38a7-4687-ae42-79dc16ab6c6a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.433766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.433743 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtzg\" (UniqueName: \"kubernetes.io/projected/95901769-38a7-4687-ae42-79dc16ab6c6a-kube-api-access-8qtzg\") pod \"payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.499519 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:01:24.499490 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:01:24.530134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.530079 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:24.613055 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.612068 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:24.654044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.653342 5272 generic.go:356] "Generic (PLEG): container finished" podID="913fcaca-efec-4e75-a731-5aade94d4831" containerID="09c390657ae68f5dd21665aaeadfde341fa451c08ed68ed92c08aada3ca13f7c" exitCode=0 Nov 03 23:01:24.654044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.653609 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" event={"ID":"913fcaca-efec-4e75-a731-5aade94d4831","Type":"ContainerDied","Data":"09c390657ae68f5dd21665aaeadfde341fa451c08ed68ed92c08aada3ca13f7c"} Nov 03 23:01:24.656345 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.656149 5272 generic.go:356] "Generic (PLEG): container finished" podID="06db652e-54ce-4405-b7af-fdcf65b8bf0b" containerID="89178ca87a5c3bee115f56205134bb817613f85e82d07bf7af7adf8e338676e6" exitCode=0 Nov 03 23:01:24.656345 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.656202 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" event={"ID":"06db652e-54ce-4405-b7af-fdcf65b8bf0b","Type":"ContainerDied","Data":"89178ca87a5c3bee115f56205134bb817613f85e82d07bf7af7adf8e338676e6"} Nov 03 23:01:24.908564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:24.908532 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz"] Nov 03 23:01:24.914222 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:24.914190 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95901769_38a7_4687_ae42_79dc16ab6c6a.slice/crio-f88e42c26e9037a08120fd402c5099c3003ad573e67f257ba23418b72f3772c9 WatchSource:0}: Error finding container f88e42c26e9037a08120fd402c5099c3003ad573e67f257ba23418b72f3772c9: Status 404 returned error can't find the container with id f88e42c26e9037a08120fd402c5099c3003ad573e67f257ba23418b72f3772c9 Nov 03 23:01:25.009062 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.009037 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4"] Nov 03 23:01:25.014567 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:25.014543 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fbef3b_965d_4a61_bebc_63f605f0b47f.slice/crio-be6a3761eb99bd231d7ca56dba8bce0c7e5af45a5c811a33a26c3934c749b682 WatchSource:0}: Error finding container be6a3761eb99bd231d7ca56dba8bce0c7e5af45a5c811a33a26c3934c749b682: Status 404 returned error can't find the container with id be6a3761eb99bd231d7ca56dba8bce0c7e5af45a5c811a33a26c3934c749b682 Nov 03 23:01:25.664559 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.664519 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" event={"ID":"95901769-38a7-4687-ae42-79dc16ab6c6a","Type":"ContainerStarted","Data":"4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7"} Nov 03 23:01:25.665033 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.664596 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" event={"ID":"95901769-38a7-4687-ae42-79dc16ab6c6a","Type":"ContainerStarted","Data":"f88e42c26e9037a08120fd402c5099c3003ad573e67f257ba23418b72f3772c9"} Nov 03 23:01:25.665033 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.664629 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:25.671002 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.670961 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" event={"ID":"14fbef3b-965d-4a61-bebc-63f605f0b47f","Type":"ContainerStarted","Data":"29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1"} Nov 03 23:01:25.671269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.671239 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:25.671506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.671284 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" event={"ID":"14fbef3b-965d-4a61-bebc-63f605f0b47f","Type":"ContainerStarted","Data":"be6a3761eb99bd231d7ca56dba8bce0c7e5af45a5c811a33a26c3934c749b682"} Nov 03 23:01:25.775003 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.774952 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" podStartSLOduration=1.774939466 podStartE2EDuration="1.774939466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:25.773290975 +0000 UTC m=+2266.395599285" watchObservedRunningTime="2026-11-03 23:01:25.774939466 +0000 UTC m=+2266.397247773" Nov 03 23:01:25.857656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.857632 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:25.860086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.860065 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.884509 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" podStartSLOduration=2.884493765 podStartE2EDuration="2.884493765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:25.882103127 +0000 UTC m=+2266.504411440" watchObservedRunningTime="2026-11-03 23:01:25.884493765 +0000 UTC m=+2266.506802074" Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.900533 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913fcaca-efec-4e75-a731-5aade94d4831-kube-api-access-4f9g4\" (UniqueName: \"kubernetes.io/projected/913fcaca-efec-4e75-a731-5aade94d4831-kube-api-access-4f9g4\") pod \"913fcaca-efec-4e75-a731-5aade94d4831\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.900592 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06db652e-54ce-4405-b7af-fdcf65b8bf0b-kube-api-access-v6mtd\" (UniqueName: \"kubernetes.io/projected/06db652e-54ce-4405-b7af-fdcf65b8bf0b-kube-api-access-v6mtd\") pod \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.900836 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-openshift-ca\") pod \"913fcaca-efec-4e75-a731-5aade94d4831\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.900900 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-xks-ca\") pod \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.901012 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-openshift-ca\") pod \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\" (UID: \"06db652e-54ce-4405-b7af-fdcf65b8bf0b\") " Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.901042 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-xks-ca\") pod \"913fcaca-efec-4e75-a731-5aade94d4831\" (UID: \"913fcaca-efec-4e75-a731-5aade94d4831\") " Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.907656 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-openshift-ca" pod "913fcaca-efec-4e75-a731-5aade94d4831" (UID: "913fcaca-efec-4e75-a731-5aade94d4831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:25.925960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.909121 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-openshift-ca" pod "06db652e-54ce-4405-b7af-fdcf65b8bf0b" (UID: "06db652e-54ce-4405-b7af-fdcf65b8bf0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:25.946166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.937688 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06db652e-54ce-4405-b7af-fdcf65b8bf0b-kube-api-access-v6mtd" pod "06db652e-54ce-4405-b7af-fdcf65b8bf0b" (UID: "06db652e-54ce-4405-b7af-fdcf65b8bf0b"). InnerVolumeSpecName "kube-api-access-v6mtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:25.946166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.937780 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-xks-ca" pod "913fcaca-efec-4e75-a731-5aade94d4831" (UID: "913fcaca-efec-4e75-a731-5aade94d4831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:25.946166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.938556 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-xks-ca" pod "06db652e-54ce-4405-b7af-fdcf65b8bf0b" (UID: "06db652e-54ce-4405-b7af-fdcf65b8bf0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:25.952863 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:25.952462 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913fcaca-efec-4e75-a731-5aade94d4831-kube-api-access-4f9g4" pod "913fcaca-efec-4e75-a731-5aade94d4831" (UID: "913fcaca-efec-4e75-a731-5aade94d4831"). InnerVolumeSpecName "kube-api-access-4f9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:26.016829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.011080 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:26.016829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.011124 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f9g4\" (UniqueName: \"kubernetes.io/projected/913fcaca-efec-4e75-a731-5aade94d4831-kube-api-access-4f9g4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:26.016829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.011141 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mtd\" (UniqueName: \"kubernetes.io/projected/06db652e-54ce-4405-b7af-fdcf65b8bf0b-kube-api-access-v6mtd\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:26.016829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.011154 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913fcaca-efec-4e75-a731-5aade94d4831-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:26.016829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.011168 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:26.016829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.011182 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06db652e-54ce-4405-b7af-fdcf65b8bf0b-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:26.679589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.679553 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" Nov 03 23:01:26.680073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.679587 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4" event={"ID":"06db652e-54ce-4405-b7af-fdcf65b8bf0b","Type":"ContainerDied","Data":"9b420ff1d5749f2cdf5fcd29df318af7b35dfceda0917a2ac94fbf99e45f5c23"} Nov 03 23:01:26.680073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.679644 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b420ff1d5749f2cdf5fcd29df318af7b35dfceda0917a2ac94fbf99e45f5c23" Nov 03 23:01:26.681025 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.681006 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" Nov 03 23:01:26.681144 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.681055 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq" event={"ID":"913fcaca-efec-4e75-a731-5aade94d4831","Type":"ContainerDied","Data":"68c53a166ca81f509704ffbc1825e65017178fe6d3f61cd976f01e3ff95a311c"} Nov 03 23:01:26.681144 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:26.681096 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c53a166ca81f509704ffbc1825e65017178fe6d3f61cd976f01e3ff95a311c" Nov 03 23:01:30.670192 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.670158 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5"] Nov 03 23:01:30.670777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.670763 5272 cpu_manager.go:414] "containerMap: removing container" podUID="913fcaca-efec-4e75-a731-5aade94d4831" containerName="revoke-keys" Nov 03 23:01:30.670889 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.670778 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913fcaca-efec-4e75-a731-5aade94d4831" containerName="revoke-keys" Nov 03 23:01:30.670889 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.670795 5272 cpu_manager.go:414] "containerMap: removing container" podUID="06db652e-54ce-4405-b7af-fdcf65b8bf0b" containerName="revoke-keys" Nov 03 23:01:30.670889 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.670815 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06db652e-54ce-4405-b7af-fdcf65b8bf0b" containerName="revoke-keys" Nov 03 23:01:30.671017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.670943 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="913fcaca-efec-4e75-a731-5aade94d4831" containerName="revoke-keys" Nov 03 23:01:30.671017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.670955 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="06db652e-54ce-4405-b7af-fdcf65b8bf0b" containerName="revoke-keys" Nov 03 23:01:30.674786 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.674759 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.774587 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.774544 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:30.774767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.774602 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:01:30.774767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.774675 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:01:30.774767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.774736 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:30.775201 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.775175 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.775273 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.775228 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q849x\" (UniqueName: \"kubernetes.io/projected/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-kube-api-access-q849x\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.775499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.775477 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.778185 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.778159 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5"] Nov 03 23:01:30.877229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.877191 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.877386 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.877280 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.877386 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.877312 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q849x\" (UniqueName: \"kubernetes.io/projected/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-kube-api-access-q849x\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.878063 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.878040 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.879629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.879606 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.934371 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.934314 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q849x\" (UniqueName: \"kubernetes.io/projected/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-kube-api-access-q849x\") pod \"maas-api-revoke-sub-116d4246-xn4w5\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:30.988420 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:30.988393 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:31.362266 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:31.362232 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5"] Nov 03 23:01:31.368686 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:31.368658 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c47846_2bf7_4bb6_a407_d31ecb75f2df.slice/crio-adff3327039d53e76be8217903f8c98efad501cb2463fd97d7759d9bf50c2f95 WatchSource:0}: Error finding container adff3327039d53e76be8217903f8c98efad501cb2463fd97d7759d9bf50c2f95: Status 404 returned error can't find the container with id adff3327039d53e76be8217903f8c98efad501cb2463fd97d7759d9bf50c2f95 Nov 03 23:01:31.719682 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:31.719642 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" event={"ID":"b7c47846-2bf7-4bb6-a407-d31ecb75f2df","Type":"ContainerStarted","Data":"c37e1342b9894b143f3d01e6c0ff01c977d7405439efe301e5674f2debf0ebc4"} Nov 03 23:01:31.720157 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:31.719712 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" event={"ID":"b7c47846-2bf7-4bb6-a407-d31ecb75f2df","Type":"ContainerStarted","Data":"adff3327039d53e76be8217903f8c98efad501cb2463fd97d7759d9bf50c2f95"} Nov 03 23:01:31.832667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:31.832614 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" podStartSLOduration=1.832598138 podStartE2EDuration="1.832598138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:31.831547397 +0000 UTC m=+2272.453855707" watchObservedRunningTime="2026-11-03 23:01:31.832598138 +0000 UTC m=+2272.454906447" Nov 03 23:01:32.634867 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:32.634838 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:01:32.635073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:32.634909 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:01:32.727532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:32.727491 5272 generic.go:356] "Generic (PLEG): container finished" podID="b7c47846-2bf7-4bb6-a407-d31ecb75f2df" containerID="c37e1342b9894b143f3d01e6c0ff01c977d7405439efe301e5674f2debf0ebc4" exitCode=0 Nov 03 23:01:32.727961 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:32.727534 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" event={"ID":"b7c47846-2bf7-4bb6-a407-d31ecb75f2df","Type":"ContainerDied","Data":"c37e1342b9894b143f3d01e6c0ff01c977d7405439efe301e5674f2debf0ebc4"} Nov 03 23:01:33.890919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:33.890891 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:34.016496 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.016412 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-kube-api-access-q849x\" (UniqueName: \"kubernetes.io/projected/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-kube-api-access-q849x\") pod \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " Nov 03 23:01:34.016666 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.016632 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-openshift-ca\") pod \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " Nov 03 23:01:34.016749 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.016725 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-xks-ca\") pod \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\" (UID: \"b7c47846-2bf7-4bb6-a407-d31ecb75f2df\") " Nov 03 23:01:34.017112 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.017069 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-openshift-ca" pod "b7c47846-2bf7-4bb6-a407-d31ecb75f2df" (UID: "b7c47846-2bf7-4bb6-a407-d31ecb75f2df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:34.017328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.017308 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:34.019171 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.019142 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-xks-ca" pod "b7c47846-2bf7-4bb6-a407-d31ecb75f2df" (UID: "b7c47846-2bf7-4bb6-a407-d31ecb75f2df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:34.019304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.019286 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-kube-api-access-q849x" pod "b7c47846-2bf7-4bb6-a407-d31ecb75f2df" (UID: "b7c47846-2bf7-4bb6-a407-d31ecb75f2df"). InnerVolumeSpecName "kube-api-access-q849x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:34.118824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.118765 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:34.118824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.118798 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q849x\" (UniqueName: \"kubernetes.io/projected/b7c47846-2bf7-4bb6-a407-d31ecb75f2df-kube-api-access-q849x\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:34.742788 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.742752 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" event={"ID":"b7c47846-2bf7-4bb6-a407-d31ecb75f2df","Type":"ContainerDied","Data":"adff3327039d53e76be8217903f8c98efad501cb2463fd97d7759d9bf50c2f95"} Nov 03 23:01:34.742974 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.742793 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5" Nov 03 23:01:34.742974 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:34.742832 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adff3327039d53e76be8217903f8c98efad501cb2463fd97d7759d9bf50c2f95" Nov 03 23:01:36.684522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:36.684492 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:01:36.685009 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:36.684564 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:01:38.444731 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.444690 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442"] Nov 03 23:01:38.445534 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.445511 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b7c47846-2bf7-4bb6-a407-d31ecb75f2df" containerName="revoke-keys" Nov 03 23:01:38.445630 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.445538 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c47846-2bf7-4bb6-a407-d31ecb75f2df" containerName="revoke-keys" Nov 03 23:01:38.445754 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.445739 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c47846-2bf7-4bb6-a407-d31ecb75f2df" containerName="revoke-keys" Nov 03 23:01:38.449535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.449513 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.502457 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.502425 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:38.502457 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.502425 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:38.506693 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.506668 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442"] Nov 03 23:01:38.549012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.548984 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:01:38.549186 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.549033 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:01:38.605187 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.605150 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.605370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.605329 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd674\" (UniqueName: \"kubernetes.io/projected/d1811472-e347-410a-bf73-7794f8ec4267-kube-api-access-wd674\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.605425 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.605374 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.706973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.706883 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd674\" (UniqueName: \"kubernetes.io/projected/d1811472-e347-410a-bf73-7794f8ec4267-kube-api-access-wd674\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.706973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.706937 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.707183 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.707020 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.707713 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.707691 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.709361 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.709341 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:38.764246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:38.764215 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd674\" (UniqueName: \"kubernetes.io/projected/d1811472-e347-410a-bf73-7794f8ec4267-kube-api-access-wd674\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:39.063292 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:39.063255 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:39.442578 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:39.442497 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442"] Nov 03 23:01:39.448077 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:39.448046 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1811472_e347_410a_bf73_7794f8ec4267.slice/crio-fc4cd3748127ee054fff5cecd6d493ab12a61ef4669ade6cf9c7fc841dc1907f WatchSource:0}: Error finding container fc4cd3748127ee054fff5cecd6d493ab12a61ef4669ade6cf9c7fc841dc1907f: Status 404 returned error can't find the container with id fc4cd3748127ee054fff5cecd6d493ab12a61ef4669ade6cf9c7fc841dc1907f Nov 03 23:01:39.779314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:39.779283 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" event={"ID":"d1811472-e347-410a-bf73-7794f8ec4267","Type":"ContainerStarted","Data":"6f6505a27d7a08cd048ccb36209d87d46e23317790aa05ddeb4d41279477c6aa"} Nov 03 23:01:39.779419 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:39.779369 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" event={"ID":"d1811472-e347-410a-bf73-7794f8ec4267","Type":"ContainerStarted","Data":"fc4cd3748127ee054fff5cecd6d493ab12a61ef4669ade6cf9c7fc841dc1907f"} Nov 03 23:01:39.890388 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:39.890337 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" podStartSLOduration=1.890323647 podStartE2EDuration="1.890323647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:39.887472453 +0000 UTC m=+2280.509780761" watchObservedRunningTime="2026-11-03 23:01:39.890323647 +0000 UTC m=+2280.512631948" Nov 03 23:01:40.788592 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:40.788554 5272 generic.go:356] "Generic (PLEG): container finished" podID="d1811472-e347-410a-bf73-7794f8ec4267" containerID="6f6505a27d7a08cd048ccb36209d87d46e23317790aa05ddeb4d41279477c6aa" exitCode=0 Nov 03 23:01:40.788985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:40.788637 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" event={"ID":"d1811472-e347-410a-bf73-7794f8ec4267","Type":"ContainerDied","Data":"6f6505a27d7a08cd048ccb36209d87d46e23317790aa05ddeb4d41279477c6aa"} Nov 03 23:01:41.928501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.928474 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv"] Nov 03 23:01:41.933049 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.933030 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:41.948357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.948336 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:41.960666 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.960642 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1811472-e347-410a-bf73-7794f8ec4267-kube-api-access-wd674\" (UniqueName: \"kubernetes.io/projected/d1811472-e347-410a-bf73-7794f8ec4267-kube-api-access-wd674\") pod \"d1811472-e347-410a-bf73-7794f8ec4267\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " Nov 03 23:01:41.960775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.960712 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-openshift-ca\") pod \"d1811472-e347-410a-bf73-7794f8ec4267\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " Nov 03 23:01:41.960775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.960733 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-xks-ca\") pod \"d1811472-e347-410a-bf73-7794f8ec4267\" (UID: \"d1811472-e347-410a-bf73-7794f8ec4267\") " Nov 03 23:01:41.961061 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.961036 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:41.961061 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.961044 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-openshift-ca" pod "d1811472-e347-410a-bf73-7794f8ec4267" (UID: "d1811472-e347-410a-bf73-7794f8ec4267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:41.961199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.961170 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhn2\" (UniqueName: \"kubernetes.io/projected/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-kube-api-access-tjhn2\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:41.961259 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.961244 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:41.961316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.961303 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:41.962725 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.962700 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1811472-e347-410a-bf73-7794f8ec4267-kube-api-access-wd674" pod "d1811472-e347-410a-bf73-7794f8ec4267" (UID: "d1811472-e347-410a-bf73-7794f8ec4267"). InnerVolumeSpecName "kube-api-access-wd674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:41.962833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.962781 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-xks-ca" pod "d1811472-e347-410a-bf73-7794f8ec4267" (UID: "d1811472-e347-410a-bf73-7794f8ec4267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:41.986478 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:41.986454 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv"] Nov 03 23:01:42.062316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.062229 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:42.062499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.062323 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:42.062499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.062406 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhn2\" (UniqueName: \"kubernetes.io/projected/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-kube-api-access-tjhn2\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:42.062499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.062496 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1811472-e347-410a-bf73-7794f8ec4267-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:42.062690 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.062513 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd674\" (UniqueName: \"kubernetes.io/projected/d1811472-e347-410a-bf73-7794f8ec4267-kube-api-access-wd674\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:42.063059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.063038 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:42.064632 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.064607 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:42.119328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.119296 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhn2\" (UniqueName: \"kubernetes.io/projected/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-kube-api-access-tjhn2\") pod \"maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:42.257404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.257375 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:42.632924 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.632841 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv"] Nov 03 23:01:42.638390 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:42.638364 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec313b2b_e4c0_460e_ac49_464b8e4f80e5.slice/crio-57a77deb7682000153e6493ae5dc29e69654214b672c0074000314cc90d43e6e WatchSource:0}: Error finding container 57a77deb7682000153e6493ae5dc29e69654214b672c0074000314cc90d43e6e: Status 404 returned error can't find the container with id 57a77deb7682000153e6493ae5dc29e69654214b672c0074000314cc90d43e6e Nov 03 23:01:42.803957 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.803921 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" Nov 03 23:01:42.804139 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.803953 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442" event={"ID":"d1811472-e347-410a-bf73-7794f8ec4267","Type":"ContainerDied","Data":"fc4cd3748127ee054fff5cecd6d493ab12a61ef4669ade6cf9c7fc841dc1907f"} Nov 03 23:01:42.804139 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.804004 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4cd3748127ee054fff5cecd6d493ab12a61ef4669ade6cf9c7fc841dc1907f" Nov 03 23:01:42.805393 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:42.805369 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" event={"ID":"ec313b2b-e4c0-460e-ac49-464b8e4f80e5","Type":"ContainerStarted","Data":"57a77deb7682000153e6493ae5dc29e69654214b672c0074000314cc90d43e6e"} Nov 03 23:01:43.223944 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:43.223867 5272 scope.go:122] "RemoveContainer" containerID="d09bc117f4dc13afba4c36604cc683499fd4225d5ffe12b0565ea36845c2d9fd" Nov 03 23:01:43.233564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:43.233543 5272 scope.go:122] "RemoveContainer" containerID="c450ed4d0ae725e5b300981290b232cba048f9911677fb22a6f0756d5c093c64" Nov 03 23:01:43.814346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:43.814309 5272 generic.go:356] "Generic (PLEG): container finished" podID="ec313b2b-e4c0-460e-ac49-464b8e4f80e5" containerID="e65c733b6d2f8086fd4570e5ceb0fdf09ed7177b77c281f5c317b97e037ecb89" exitCode=0 Nov 03 23:01:43.814535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:43.814399 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" event={"ID":"ec313b2b-e4c0-460e-ac49-464b8e4f80e5","Type":"ContainerDied","Data":"e65c733b6d2f8086fd4570e5ceb0fdf09ed7177b77c281f5c317b97e037ecb89"} Nov 03 23:01:44.981658 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:44.981633 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:45.098613 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.098518 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-openshift-ca\") pod \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " Nov 03 23:01:45.098995 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.098964 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-kube-api-access-tjhn2\" (UniqueName: \"kubernetes.io/projected/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-kube-api-access-tjhn2\") pod \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " Nov 03 23:01:45.099376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.099098 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-openshift-ca" pod "ec313b2b-e4c0-460e-ac49-464b8e4f80e5" (UID: "ec313b2b-e4c0-460e-ac49-464b8e4f80e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:45.099527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.099508 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-xks-ca\") pod \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\" (UID: \"ec313b2b-e4c0-460e-ac49-464b8e4f80e5\") " Nov 03 23:01:45.100056 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.100028 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:45.101409 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.101377 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-kube-api-access-tjhn2" pod "ec313b2b-e4c0-460e-ac49-464b8e4f80e5" (UID: "ec313b2b-e4c0-460e-ac49-464b8e4f80e5"). InnerVolumeSpecName "kube-api-access-tjhn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:45.101700 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.101674 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-xks-ca" pod "ec313b2b-e4c0-460e-ac49-464b8e4f80e5" (UID: "ec313b2b-e4c0-460e-ac49-464b8e4f80e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:45.201106 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.201068 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjhn2\" (UniqueName: \"kubernetes.io/projected/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-kube-api-access-tjhn2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:45.201106 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.201107 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec313b2b-e4c0-460e-ac49-464b8e4f80e5-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:45.829182 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.829102 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" Nov 03 23:01:45.829328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.829102 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv" event={"ID":"ec313b2b-e4c0-460e-ac49-464b8e4f80e5","Type":"ContainerDied","Data":"57a77deb7682000153e6493ae5dc29e69654214b672c0074000314cc90d43e6e"} Nov 03 23:01:45.829328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:45.829225 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57a77deb7682000153e6493ae5dc29e69654214b672c0074000314cc90d43e6e" Nov 03 23:01:49.148943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.148909 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw"] Nov 03 23:01:49.149731 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.149709 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ec313b2b-e4c0-460e-ac49-464b8e4f80e5" containerName="revoke-keys" Nov 03 23:01:49.149841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.149732 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec313b2b-e4c0-460e-ac49-464b8e4f80e5" containerName="revoke-keys" Nov 03 23:01:49.149841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.149769 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d1811472-e347-410a-bf73-7794f8ec4267" containerName="revoke-keys" Nov 03 23:01:49.149841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.149777 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1811472-e347-410a-bf73-7794f8ec4267" containerName="revoke-keys" Nov 03 23:01:49.149994 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.149977 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec313b2b-e4c0-460e-ac49-464b8e4f80e5" containerName="revoke-keys" Nov 03 23:01:49.150046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.150000 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1811472-e347-410a-bf73-7794f8ec4267" containerName="revoke-keys" Nov 03 23:01:49.153603 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.153586 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.202581 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.202548 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:01:49.248817 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.248772 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:49.248990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.248973 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:01:49.249097 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.249064 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:49.251871 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.251847 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.251990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.251898 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4bqb\" (UniqueName: \"kubernetes.io/projected/263c14ec-4fdf-48c0-87ac-2330164d2b92-kube-api-access-x4bqb\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.252035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.252009 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.253915 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.253893 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw"] Nov 03 23:01:49.353558 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.353524 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.353710 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.353565 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4bqb\" (UniqueName: \"kubernetes.io/projected/263c14ec-4fdf-48c0-87ac-2330164d2b92-kube-api-access-x4bqb\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.353710 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.353606 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.354282 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.354258 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.355899 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.355882 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.410684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.410627 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4bqb\" (UniqueName: \"kubernetes.io/projected/263c14ec-4fdf-48c0-87ac-2330164d2b92-kube-api-access-x4bqb\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.467669 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.467647 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:49.844497 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.844463 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw"] Nov 03 23:01:49.850140 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:01:49.850115 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263c14ec_4fdf_48c0_87ac_2330164d2b92.slice/crio-ee7b13ab9da28f55c475648a692458959d3738b78f46041ac5bfe7ca708164c1 WatchSource:0}: Error finding container ee7b13ab9da28f55c475648a692458959d3738b78f46041ac5bfe7ca708164c1: Status 404 returned error can't find the container with id ee7b13ab9da28f55c475648a692458959d3738b78f46041ac5bfe7ca708164c1 Nov 03 23:01:49.860199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:49.860172 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" event={"ID":"263c14ec-4fdf-48c0-87ac-2330164d2b92","Type":"ContainerStarted","Data":"ee7b13ab9da28f55c475648a692458959d3738b78f46041ac5bfe7ca708164c1"} Nov 03 23:01:50.868888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:50.868859 5272 generic.go:356] "Generic (PLEG): container finished" podID="263c14ec-4fdf-48c0-87ac-2330164d2b92" containerID="9c49bdedca283344ee99c4f028528c431e5ca9f77a88c4139aaa8bce31e0a035" exitCode=0 Nov 03 23:01:50.869270 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:50.868894 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" event={"ID":"263c14ec-4fdf-48c0-87ac-2330164d2b92","Type":"ContainerDied","Data":"9c49bdedca283344ee99c4f028528c431e5ca9f77a88c4139aaa8bce31e0a035"} Nov 03 23:01:51.378229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:51.378190 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz"] Nov 03 23:01:51.483751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:51.483716 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk"] Nov 03 23:01:52.093865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.093842 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:52.197798 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.197708 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263c14ec-4fdf-48c0-87ac-2330164d2b92-kube-api-access-x4bqb\" (UniqueName: \"kubernetes.io/projected/263c14ec-4fdf-48c0-87ac-2330164d2b92-kube-api-access-x4bqb\") pod \"263c14ec-4fdf-48c0-87ac-2330164d2b92\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " Nov 03 23:01:52.198011 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.197862 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-xks-ca\") pod \"263c14ec-4fdf-48c0-87ac-2330164d2b92\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " Nov 03 23:01:52.198011 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.197985 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-openshift-ca\") pod \"263c14ec-4fdf-48c0-87ac-2330164d2b92\" (UID: \"263c14ec-4fdf-48c0-87ac-2330164d2b92\") " Nov 03 23:01:52.198409 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.198378 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-openshift-ca" pod "263c14ec-4fdf-48c0-87ac-2330164d2b92" (UID: "263c14ec-4fdf-48c0-87ac-2330164d2b92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:52.200060 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.200034 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-xks-ca" pod "263c14ec-4fdf-48c0-87ac-2330164d2b92" (UID: "263c14ec-4fdf-48c0-87ac-2330164d2b92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:52.200173 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.200127 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263c14ec-4fdf-48c0-87ac-2330164d2b92-kube-api-access-x4bqb" pod "263c14ec-4fdf-48c0-87ac-2330164d2b92" (UID: "263c14ec-4fdf-48c0-87ac-2330164d2b92"). InnerVolumeSpecName "kube-api-access-x4bqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:52.299377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.299338 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:52.299377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.299372 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263c14ec-4fdf-48c0-87ac-2330164d2b92-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:52.299377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.299384 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4bqb\" (UniqueName: \"kubernetes.io/projected/263c14ec-4fdf-48c0-87ac-2330164d2b92-kube-api-access-x4bqb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:52.928830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.928634 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" event={"ID":"263c14ec-4fdf-48c0-87ac-2330164d2b92","Type":"ContainerDied","Data":"ee7b13ab9da28f55c475648a692458959d3738b78f46041ac5bfe7ca708164c1"} Nov 03 23:01:52.928830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.928718 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee7b13ab9da28f55c475648a692458959d3738b78f46041ac5bfe7ca708164c1" Nov 03 23:01:52.929100 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:52.928871 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw" Nov 03 23:01:53.528524 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:53.528487 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz"] Nov 03 23:01:53.574597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:53.574547 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb"] Nov 03 23:01:53.578528 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:53.578496 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499"] Nov 03 23:01:53.903681 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:53.903643 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb"] Nov 03 23:01:54.011920 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:54.011853 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc"] Nov 03 23:01:54.511716 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:01:54.511687 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:01:56.379129 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.379088 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" podUID="72d94103-8548-4474-93d1-bc5a003d53b4" containerName="payload-processing" containerID="cri-o://ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6" gracePeriod=25 Nov 03 23:01:56.484936 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.484883 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" podUID="dac539b8-4b24-4c8e-ab9c-aabaa943b303" containerName="payload-pre-processing" containerID="cri-o://b0d1d7aaad50b2f7e6206f0b9ba54090d5a025ec795ceb109cd0c4ddfc0cb451" gracePeriod=25 Nov 03 23:01:56.863705 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.863673 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 23:01:56.957026 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.956994 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d94103-8548-4474-93d1-bc5a003d53b4-kube-api-access-bmz56\" (UniqueName: \"kubernetes.io/projected/72d94103-8548-4474-93d1-bc5a003d53b4-kube-api-access-bmz56\") pod \"72d94103-8548-4474-93d1-bc5a003d53b4\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " Nov 03 23:01:56.957191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.957141 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d94103-8548-4474-93d1-bc5a003d53b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d94103-8548-4474-93d1-bc5a003d53b4-plugins-config-volume\") pod \"72d94103-8548-4474-93d1-bc5a003d53b4\" (UID: \"72d94103-8548-4474-93d1-bc5a003d53b4\") " Nov 03 23:01:56.957589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.957559 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d94103-8548-4474-93d1-bc5a003d53b4-plugins-config-volume" pod "72d94103-8548-4474-93d1-bc5a003d53b4" (UID: "72d94103-8548-4474-93d1-bc5a003d53b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:56.957688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.957649 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d94103-8548-4474-93d1-bc5a003d53b4-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:56.959221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.959194 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d94103-8548-4474-93d1-bc5a003d53b4-kube-api-access-bmz56" pod "72d94103-8548-4474-93d1-bc5a003d53b4" (UID: "72d94103-8548-4474-93d1-bc5a003d53b4"). InnerVolumeSpecName "kube-api-access-bmz56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:56.961836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.961788 5272 generic.go:356] "Generic (PLEG): container finished" podID="dac539b8-4b24-4c8e-ab9c-aabaa943b303" containerID="b0d1d7aaad50b2f7e6206f0b9ba54090d5a025ec795ceb109cd0c4ddfc0cb451" exitCode=0 Nov 03 23:01:56.961943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.961835 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" event={"ID":"dac539b8-4b24-4c8e-ab9c-aabaa943b303","Type":"ContainerDied","Data":"b0d1d7aaad50b2f7e6206f0b9ba54090d5a025ec795ceb109cd0c4ddfc0cb451"} Nov 03 23:01:56.963751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.963603 5272 generic.go:356] "Generic (PLEG): container finished" podID="72d94103-8548-4474-93d1-bc5a003d53b4" containerID="ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6" exitCode=0 Nov 03 23:01:56.963751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.963722 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" Nov 03 23:01:56.964250 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.964149 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" event={"ID":"72d94103-8548-4474-93d1-bc5a003d53b4","Type":"ContainerDied","Data":"ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6"} Nov 03 23:01:56.964250 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.964209 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz" event={"ID":"72d94103-8548-4474-93d1-bc5a003d53b4","Type":"ContainerDied","Data":"646128cfd64a9eb450fb2552e8b63bcce1138182ee3acb75649f35f1545eb998"} Nov 03 23:01:56.964250 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.964239 5272 scope.go:122] "RemoveContainer" containerID="ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6" Nov 03 23:01:56.980532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.980482 5272 scope.go:122] "RemoveContainer" containerID="ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6" Nov 03 23:01:56.980843 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:01:56.980793 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6\": container with ID starting with ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6 not found: ID does not exist" containerID="ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6" Nov 03 23:01:56.980985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.980850 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6"} err="failed to get container status \"ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6\": rpc error: code = NotFound desc = could not find container \"ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6\": container with ID starting with ed7cd399923178e639db8776e3f0c4c3da55c5e215218db54d3f39efb11ab4f6 not found: ID does not exist" Nov 03 23:01:56.983595 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:56.983574 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 23:01:57.058456 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.058421 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac539b8-4b24-4c8e-ab9c-aabaa943b303-kube-api-access-tz8rn\" (UniqueName: \"kubernetes.io/projected/dac539b8-4b24-4c8e-ab9c-aabaa943b303-kube-api-access-tz8rn\") pod \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " Nov 03 23:01:57.058635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.058520 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac539b8-4b24-4c8e-ab9c-aabaa943b303-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac539b8-4b24-4c8e-ab9c-aabaa943b303-plugins-config-volume\") pod \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\" (UID: \"dac539b8-4b24-4c8e-ab9c-aabaa943b303\") " Nov 03 23:01:57.058881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.058863 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmz56\" (UniqueName: \"kubernetes.io/projected/72d94103-8548-4474-93d1-bc5a003d53b4-kube-api-access-bmz56\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:57.059025 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.058998 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac539b8-4b24-4c8e-ab9c-aabaa943b303-plugins-config-volume" pod "dac539b8-4b24-4c8e-ab9c-aabaa943b303" (UID: "dac539b8-4b24-4c8e-ab9c-aabaa943b303"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:57.060512 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.060495 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac539b8-4b24-4c8e-ab9c-aabaa943b303-kube-api-access-tz8rn" pod "dac539b8-4b24-4c8e-ab9c-aabaa943b303" (UID: "dac539b8-4b24-4c8e-ab9c-aabaa943b303"). InnerVolumeSpecName "kube-api-access-tz8rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:57.133744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.133704 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz"] Nov 03 23:01:57.138221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.138188 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a03fee-5d9bb89966-kfsbz"] Nov 03 23:01:57.159733 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.159700 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8rn\" (UniqueName: \"kubernetes.io/projected/dac539b8-4b24-4c8e-ab9c-aabaa943b303-kube-api-access-tz8rn\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:57.159733 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.159728 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac539b8-4b24-4c8e-ab9c-aabaa943b303-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:57.494673 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.494639 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d94103-8548-4474-93d1-bc5a003d53b4" path="/var/lib/kubelet/pods/72d94103-8548-4474-93d1-bc5a003d53b4/volumes" Nov 03 23:01:57.982829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.982419 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" event={"ID":"dac539b8-4b24-4c8e-ab9c-aabaa943b303","Type":"ContainerDied","Data":"902419a68fdfbc5c3f6942c930833b68e061f24ecbf474d4aa2086e9a9cb0e95"} Nov 03 23:01:57.982829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.982499 5272 scope.go:122] "RemoveContainer" containerID="b0d1d7aaad50b2f7e6206f0b9ba54090d5a025ec795ceb109cd0c4ddfc0cb451" Nov 03 23:01:57.982829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:57.982636 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk" Nov 03 23:01:58.140650 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:58.140613 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk"] Nov 03 23:01:58.148826 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:58.147304 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a03fee-5595ccdb5f-xgxjk"] Nov 03 23:01:58.908829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:58.905750 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" podUID="9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" containerName="payload-processing" containerID="cri-o://adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b" gracePeriod=25 Nov 03 23:01:59.014922 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.014872 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" podUID="e3dd6fdc-d6ea-4c02-83ec-53200433861f" containerName="payload-pre-processing" containerID="cri-o://b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819" gracePeriod=25 Nov 03 23:01:59.538672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.538638 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac539b8-4b24-4c8e-ab9c-aabaa943b303" path="/var/lib/kubelet/pods/dac539b8-4b24-4c8e-ab9c-aabaa943b303/volumes" Nov 03 23:01:59.542047 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.542020 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 23:01:59.544159 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.544127 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-plugins-config-volume\") pod \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " Nov 03 23:01:59.544314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.544187 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-kube-api-access-8ltxw\" (UniqueName: \"kubernetes.io/projected/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-kube-api-access-8ltxw\") pod \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\" (UID: \"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9\") " Nov 03 23:01:59.544797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.544771 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-plugins-config-volume" pod "9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" (UID: "9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:59.554643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.554613 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-kube-api-access-8ltxw" pod "9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" (UID: "9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9"). InnerVolumeSpecName "kube-api-access-8ltxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:59.570314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.570281 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 23:01:59.645352 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.645305 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3dd6fdc-d6ea-4c02-83ec-53200433861f-kube-api-access-hmr7h\" (UniqueName: \"kubernetes.io/projected/e3dd6fdc-d6ea-4c02-83ec-53200433861f-kube-api-access-hmr7h\") pod \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " Nov 03 23:01:59.645553 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.645508 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3dd6fdc-d6ea-4c02-83ec-53200433861f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3dd6fdc-d6ea-4c02-83ec-53200433861f-plugins-config-volume\") pod \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\" (UID: \"e3dd6fdc-d6ea-4c02-83ec-53200433861f\") " Nov 03 23:01:59.645870 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.645843 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3dd6fdc-d6ea-4c02-83ec-53200433861f-plugins-config-volume" pod "e3dd6fdc-d6ea-4c02-83ec-53200433861f" (UID: "e3dd6fdc-d6ea-4c02-83ec-53200433861f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:59.646267 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.646248 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3dd6fdc-d6ea-4c02-83ec-53200433861f-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:59.646267 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.646268 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:59.646417 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.646277 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltxw\" (UniqueName: \"kubernetes.io/projected/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9-kube-api-access-8ltxw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:01:59.648211 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.648034 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3dd6fdc-d6ea-4c02-83ec-53200433861f-kube-api-access-hmr7h" pod "e3dd6fdc-d6ea-4c02-83ec-53200433861f" (UID: "e3dd6fdc-d6ea-4c02-83ec-53200433861f"). InnerVolumeSpecName "kube-api-access-hmr7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:59.749074 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:01:59.749038 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmr7h\" (UniqueName: \"kubernetes.io/projected/e3dd6fdc-d6ea-4c02-83ec-53200433861f-kube-api-access-hmr7h\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:00.036225 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.036183 5272 generic.go:356] "Generic (PLEG): container finished" podID="e3dd6fdc-d6ea-4c02-83ec-53200433861f" containerID="b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819" exitCode=0 Nov 03 23:02:00.036692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.036251 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" Nov 03 23:02:00.036692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.036265 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" event={"ID":"e3dd6fdc-d6ea-4c02-83ec-53200433861f","Type":"ContainerDied","Data":"b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819"} Nov 03 23:02:00.036692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.036366 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc" event={"ID":"e3dd6fdc-d6ea-4c02-83ec-53200433861f","Type":"ContainerDied","Data":"e8151dcd22d4484bc698c5bacf398d069eedd5de443a4eefa383092835b3efee"} Nov 03 23:02:00.036692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.036401 5272 scope.go:122] "RemoveContainer" containerID="b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819" Nov 03 23:02:00.037957 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.037934 5272 generic.go:356] "Generic (PLEG): container finished" podID="9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" containerID="adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b" exitCode=0 Nov 03 23:02:00.038066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.037962 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" event={"ID":"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9","Type":"ContainerDied","Data":"adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b"} Nov 03 23:02:00.038066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.037978 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" Nov 03 23:02:00.038066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.038013 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb" event={"ID":"9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9","Type":"ContainerDied","Data":"967c138165fb8667cfd9a4140d70a205127239694123c0b84f3cbfec63dbe939"} Nov 03 23:02:00.049699 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.049672 5272 scope.go:122] "RemoveContainer" containerID="b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819" Nov 03 23:02:00.050674 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:02:00.050189 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819\": container with ID starting with b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819 not found: ID does not exist" containerID="b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819" Nov 03 23:02:00.050674 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.050228 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819"} err="failed to get container status \"b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819\": rpc error: code = NotFound desc = could not find container \"b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819\": container with ID starting with b0d6ec520abff4d2b692bac8f39b76fcc97f656ab986a992207c2ba886c4d819 not found: ID does not exist" Nov 03 23:02:00.050674 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.050249 5272 scope.go:122] "RemoveContainer" containerID="adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b" Nov 03 23:02:00.062246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.062222 5272 scope.go:122] "RemoveContainer" containerID="adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b" Nov 03 23:02:00.062559 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:02:00.062527 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b\": container with ID starting with adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b not found: ID does not exist" containerID="adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b" Nov 03 23:02:00.062674 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.062565 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b"} err="failed to get container status \"adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b\": rpc error: code = NotFound desc = could not find container \"adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b\": container with ID starting with adcedacc696ecd8432551051cb24cde67de24c75aaf9122397cba85bf7546b8b not found: ID does not exist" Nov 03 23:02:00.198136 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.198032 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc"] Nov 03 23:02:00.242283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.242248 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-545b5d-5c89cdd65f-w2cnc"] Nov 03 23:02:00.306539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.306506 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb"] Nov 03 23:02:00.312400 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:00.312374 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-545b5d-6ff8c4558-75gnb"] Nov 03 23:02:01.248691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:01.248661 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss"] Nov 03 23:02:01.362731 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:01.362700 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5"] Nov 03 23:02:01.494123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:01.494093 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" path="/var/lib/kubelet/pods/9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9/volumes" Nov 03 23:02:01.494541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:01.494527 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3dd6fdc-d6ea-4c02-83ec-53200433861f" path="/var/lib/kubelet/pods/e3dd6fdc-d6ea-4c02-83ec-53200433861f/volumes" Nov 03 23:02:06.162095 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162061 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls"] Nov 03 23:02:06.162618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162605 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" containerName="payload-processing" Nov 03 23:02:06.162670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162619 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" containerName="payload-processing" Nov 03 23:02:06.162670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162638 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e3dd6fdc-d6ea-4c02-83ec-53200433861f" containerName="payload-pre-processing" Nov 03 23:02:06.162670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162643 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3dd6fdc-d6ea-4c02-83ec-53200433861f" containerName="payload-pre-processing" Nov 03 23:02:06.162670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162660 5272 cpu_manager.go:414] "containerMap: removing container" podUID="263c14ec-4fdf-48c0-87ac-2330164d2b92" containerName="revoke-keys" Nov 03 23:02:06.162670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162665 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263c14ec-4fdf-48c0-87ac-2330164d2b92" containerName="revoke-keys" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162688 5272 cpu_manager.go:414] "containerMap: removing container" podUID="dac539b8-4b24-4c8e-ab9c-aabaa943b303" containerName="payload-pre-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162693 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac539b8-4b24-4c8e-ab9c-aabaa943b303" containerName="payload-pre-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162713 5272 cpu_manager.go:414] "containerMap: removing container" podUID="72d94103-8548-4474-93d1-bc5a003d53b4" containerName="payload-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162717 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d94103-8548-4474-93d1-bc5a003d53b4" containerName="payload-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162796 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac539b8-4b24-4c8e-ab9c-aabaa943b303" containerName="payload-pre-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162819 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d94103-8548-4474-93d1-bc5a003d53b4" containerName="payload-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162827 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b59e60a-6dfb-45ce-acb1-4a4471c8a2c9" containerName="payload-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162835 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3dd6fdc-d6ea-4c02-83ec-53200433861f" containerName="payload-pre-processing" Nov 03 23:02:06.162940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.162843 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="263c14ec-4fdf-48c0-87ac-2330164d2b92" containerName="revoke-keys" Nov 03 23:02:06.166245 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.166225 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.215874 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.215849 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:06.216792 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.216768 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76"] Nov 03 23:02:06.220918 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.220901 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.249008 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.248974 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" podUID="fe803d4b-d693-4f97-9f1c-c495aaea0276" containerName="payload-processing" containerID="cri-o://dfaa09cfebcb16909b13d36558bce92d205ac645d57ae3f608d079e861d24b88" gracePeriod=25 Nov 03 23:02:06.249843 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.249814 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-kube-api-access-ht92t\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.249976 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.249906 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.250115 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.250089 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.260109 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.260083 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:02:06.263035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.263011 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:02:06.263262 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.263243 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:02:06.267725 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.267700 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls"] Nov 03 23:02:06.275195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.274958 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76"] Nov 03 23:02:06.351434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.351399 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbjl\" (UniqueName: \"kubernetes.io/projected/d835ff5a-6984-4024-8234-f5e7e20bec9f-kube-api-access-clbjl\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.351434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.351442 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.351712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.351463 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-kube-api-access-ht92t\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.351712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.351566 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.351712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.351638 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.351922 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.351843 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.352235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.352216 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.354102 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.354084 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.363235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.363193 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" podUID="299f09cd-663e-4790-8f75-8f1f2698571c" containerName="payload-pre-processing" containerID="cri-o://559668611da7931d5e7aa5ccd1085cae651bd1030a0bee233be184a56b399d21" gracePeriod=25 Nov 03 23:02:06.391568 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.391543 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq"] Nov 03 23:02:06.395870 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.395852 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.408728 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.408708 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-kube-api-access-ht92t\") pod \"maas-api-revoke-sub-b4b7d52d-52sls\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.446489 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.446431 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb"] Nov 03 23:02:06.451455 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.451438 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq"] Nov 03 23:02:06.451548 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.451514 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.452843 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.452777 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.452973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.452952 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.453076 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.453060 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbjl\" (UniqueName: \"kubernetes.io/projected/d835ff5a-6984-4024-8234-f5e7e20bec9f-kube-api-access-clbjl\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.453145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.453090 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px2hr\" (UniqueName: \"kubernetes.io/projected/8997735e-2b8b-4d54-87b4-d881a374c7e1-kube-api-access-px2hr\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.453145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.453113 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.453248 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.453172 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.453914 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.453894 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.455730 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.455685 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.479636 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.479613 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:06.506225 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.506197 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb"] Nov 03 23:02:06.511236 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.511209 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbjl\" (UniqueName: \"kubernetes.io/projected/d835ff5a-6984-4024-8234-f5e7e20bec9f-kube-api-access-clbjl\") pod \"maas-api-revoke-sub-4377ebff-l2j76\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.534706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.534670 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:06.555750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.555260 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.555750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.555344 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.555750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.555390 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfm8\" (UniqueName: \"kubernetes.io/projected/084a5e7e-1c6a-4342-94e1-0c7517486c1b-kube-api-access-6pfm8\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.555750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.555418 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.555750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.555476 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.555750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.555558 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px2hr\" (UniqueName: \"kubernetes.io/projected/8997735e-2b8b-4d54-87b4-d881a374c7e1-kube-api-access-px2hr\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.557239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.557107 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.561741 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.561711 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.617074 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.616991 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px2hr\" (UniqueName: \"kubernetes.io/projected/8997735e-2b8b-4d54-87b4-d881a374c7e1-kube-api-access-px2hr\") pod \"maas-api-revoke-sub-37f65e2e-ftxhq\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.656516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.656482 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.656656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.656579 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfm8\" (UniqueName: \"kubernetes.io/projected/084a5e7e-1c6a-4342-94e1-0c7517486c1b-kube-api-access-6pfm8\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.656656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.656624 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.657469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.657436 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.658763 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.658742 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.709651 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.709598 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:06.713685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.713664 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfm8\" (UniqueName: \"kubernetes.io/projected/084a5e7e-1c6a-4342-94e1-0c7517486c1b-kube-api-access-6pfm8\") pod \"maas-api-revoke-sub-13be7e9e-qcdjb\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.782162 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.779771 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:06.865625 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.865593 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls"] Nov 03 23:02:06.871140 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:06.871108 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6db47f_e8c1_4ed8_8a31_bbc28e01e958.slice/crio-23fa4e401c015880b18b7f9b15f704259732e7fea4d971b08a5b90a8ca1f18be WatchSource:0}: Error finding container 23fa4e401c015880b18b7f9b15f704259732e7fea4d971b08a5b90a8ca1f18be: Status 404 returned error can't find the container with id 23fa4e401c015880b18b7f9b15f704259732e7fea4d971b08a5b90a8ca1f18be Nov 03 23:02:06.935459 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:06.935429 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76"] Nov 03 23:02:06.940730 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:06.940699 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd835ff5a_6984_4024_8234_f5e7e20bec9f.slice/crio-b3ebf876f679bb3344807b7bce58fe7e6b0cbebcc7456794c6b1e641bc51260a WatchSource:0}: Error finding container b3ebf876f679bb3344807b7bce58fe7e6b0cbebcc7456794c6b1e641bc51260a: Status 404 returned error can't find the container with id b3ebf876f679bb3344807b7bce58fe7e6b0cbebcc7456794c6b1e641bc51260a Nov 03 23:02:07.086576 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:07.086549 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq"] Nov 03 23:02:07.091791 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:07.091760 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8997735e_2b8b_4d54_87b4_d881a374c7e1.slice/crio-474bd32feee19c883abca7d20270a1833cc24a81f5b06720925bcbd2f60dc57d WatchSource:0}: Error finding container 474bd32feee19c883abca7d20270a1833cc24a81f5b06720925bcbd2f60dc57d: Status 404 returned error can't find the container with id 474bd32feee19c883abca7d20270a1833cc24a81f5b06720925bcbd2f60dc57d Nov 03 23:02:07.179501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:07.179458 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb"] Nov 03 23:02:07.184532 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:07.184500 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084a5e7e_1c6a_4342_94e1_0c7517486c1b.slice/crio-9c15f39b4bfb7ba24840dee3a07f65f93251fc599b18377886a543fa88527459 WatchSource:0}: Error finding container 9c15f39b4bfb7ba24840dee3a07f65f93251fc599b18377886a543fa88527459: Status 404 returned error can't find the container with id 9c15f39b4bfb7ba24840dee3a07f65f93251fc599b18377886a543fa88527459 Nov 03 23:02:07.186625 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:07.186591 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" event={"ID":"8997735e-2b8b-4d54-87b4-d881a374c7e1","Type":"ContainerStarted","Data":"474bd32feee19c883abca7d20270a1833cc24a81f5b06720925bcbd2f60dc57d"} Nov 03 23:02:07.188005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:07.187979 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" event={"ID":"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958","Type":"ContainerStarted","Data":"23fa4e401c015880b18b7f9b15f704259732e7fea4d971b08a5b90a8ca1f18be"} Nov 03 23:02:07.189486 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:07.189454 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" event={"ID":"d835ff5a-6984-4024-8234-f5e7e20bec9f","Type":"ContainerStarted","Data":"b3ebf876f679bb3344807b7bce58fe7e6b0cbebcc7456794c6b1e641bc51260a"} Nov 03 23:02:08.200158 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.200122 5272 generic.go:356] "Generic (PLEG): container finished" podID="d835ff5a-6984-4024-8234-f5e7e20bec9f" containerID="55b867b0929401184ccfa846382acc5ccceaaecd9b7d727fb2b187cc2f90ef26" exitCode=0 Nov 03 23:02:08.200571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.200210 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" event={"ID":"d835ff5a-6984-4024-8234-f5e7e20bec9f","Type":"ContainerDied","Data":"55b867b0929401184ccfa846382acc5ccceaaecd9b7d727fb2b187cc2f90ef26"} Nov 03 23:02:08.201948 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.201926 5272 generic.go:356] "Generic (PLEG): container finished" podID="8997735e-2b8b-4d54-87b4-d881a374c7e1" containerID="0ac5a69b19f9387740b8f517a3c702a64bf6eee5bbdf3b293bb7959f5181fe4e" exitCode=0 Nov 03 23:02:08.202069 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.202009 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" event={"ID":"8997735e-2b8b-4d54-87b4-d881a374c7e1","Type":"ContainerDied","Data":"0ac5a69b19f9387740b8f517a3c702a64bf6eee5bbdf3b293bb7959f5181fe4e"} Nov 03 23:02:08.203556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.203537 5272 generic.go:356] "Generic (PLEG): container finished" podID="2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" containerID="a8573513f4c82486e5bcf8dd9841764448507392e6c714023f75ff2f05b32005" exitCode=0 Nov 03 23:02:08.203646 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.203602 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" event={"ID":"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958","Type":"ContainerDied","Data":"a8573513f4c82486e5bcf8dd9841764448507392e6c714023f75ff2f05b32005"} Nov 03 23:02:08.205212 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.205190 5272 generic.go:356] "Generic (PLEG): container finished" podID="084a5e7e-1c6a-4342-94e1-0c7517486c1b" containerID="d86e19123f70d1c1abb640f2f1481d39549fa804582029e1133db0fc45653417" exitCode=0 Nov 03 23:02:08.205314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.205262 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" event={"ID":"084a5e7e-1c6a-4342-94e1-0c7517486c1b","Type":"ContainerDied","Data":"d86e19123f70d1c1abb640f2f1481d39549fa804582029e1133db0fc45653417"} Nov 03 23:02:08.205314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.205305 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" event={"ID":"084a5e7e-1c6a-4342-94e1-0c7517486c1b","Type":"ContainerStarted","Data":"9c15f39b4bfb7ba24840dee3a07f65f93251fc599b18377886a543fa88527459"} Nov 03 23:02:08.551157 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.551120 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" containerID="cri-o://3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae" gracePeriod=45 Nov 03 23:02:08.595950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.595918 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" containerID="cri-o://a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804" gracePeriod=45 Nov 03 23:02:08.598545 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.598511 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" containerID="cri-o://3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb" gracePeriod=45 Nov 03 23:02:08.600078 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.600057 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": dial tcp 10.133.0.85:8000: connect: connection refused" Nov 03 23:02:08.921573 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.921539 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 23:02:08.923491 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.923470 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 23:02:08.925571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.925556 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 23:02:08.986981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.986953 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-home\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-home\") pod \"17e875ca-63d0-4219-af46-5247b25c98db\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " Nov 03 23:02:08.987165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.986998 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-tmp-dir\") pod \"08311447-fe46-43b3-aa87-86f1fa65136b\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " Nov 03 23:02:08.987165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987028 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-model-cache\") pod \"a5042eed-9361-438a-a670-8b913eeac140\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " Nov 03 23:02:08.987165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987072 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5042eed-9361-438a-a670-8b913eeac140-kube-api-access-t6d2v\" (UniqueName: \"kubernetes.io/projected/a5042eed-9361-438a-a670-8b913eeac140-kube-api-access-t6d2v\") pod \"a5042eed-9361-438a-a670-8b913eeac140\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " Nov 03 23:02:08.987165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987138 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e875ca-63d0-4219-af46-5247b25c98db-tls-certs\" (UniqueName: \"kubernetes.io/secret/17e875ca-63d0-4219-af46-5247b25c98db-tls-certs\") pod \"17e875ca-63d0-4219-af46-5247b25c98db\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " Nov 03 23:02:08.987395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987180 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5042eed-9361-438a-a670-8b913eeac140-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5042eed-9361-438a-a670-8b913eeac140-tls-certs\") pod \"a5042eed-9361-438a-a670-8b913eeac140\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " Nov 03 23:02:08.987395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987262 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-tmp-dir" pod "08311447-fe46-43b3-aa87-86f1fa65136b" (UID: "08311447-fe46-43b3-aa87-86f1fa65136b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.987395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987263 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-home" pod "17e875ca-63d0-4219-af46-5247b25c98db" (UID: "17e875ca-63d0-4219-af46-5247b25c98db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.987565 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987392 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-model-cache" pod "a5042eed-9361-438a-a670-8b913eeac140" (UID: "a5042eed-9361-438a-a670-8b913eeac140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.987565 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987490 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-model-cache" pod "08311447-fe46-43b3-aa87-86f1fa65136b" (UID: "08311447-fe46-43b3-aa87-86f1fa65136b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.987677 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987218 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-model-cache\") pod \"08311447-fe46-43b3-aa87-86f1fa65136b\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " Nov 03 23:02:08.987677 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987656 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08311447-fe46-43b3-aa87-86f1fa65136b-kube-api-access-8cnk2\" (UniqueName: \"kubernetes.io/projected/08311447-fe46-43b3-aa87-86f1fa65136b-kube-api-access-8cnk2\") pod \"08311447-fe46-43b3-aa87-86f1fa65136b\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " Nov 03 23:02:08.987798 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987687 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e875ca-63d0-4219-af46-5247b25c98db-kube-api-access-7rwh8\" (UniqueName: \"kubernetes.io/projected/17e875ca-63d0-4219-af46-5247b25c98db-kube-api-access-7rwh8\") pod \"17e875ca-63d0-4219-af46-5247b25c98db\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " Nov 03 23:02:08.987798 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.987746 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-home\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-home\") pod \"08311447-fe46-43b3-aa87-86f1fa65136b\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " Nov 03 23:02:08.988173 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988118 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-home" pod "08311447-fe46-43b3-aa87-86f1fa65136b" (UID: "08311447-fe46-43b3-aa87-86f1fa65136b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.988173 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988130 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-dshm\") pod \"08311447-fe46-43b3-aa87-86f1fa65136b\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " Nov 03 23:02:08.988351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988196 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-model-cache\") pod \"17e875ca-63d0-4219-af46-5247b25c98db\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " Nov 03 23:02:08.988351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988235 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-tmp-dir\") pod \"a5042eed-9361-438a-a670-8b913eeac140\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " Nov 03 23:02:08.988351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988283 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-home\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-home\") pod \"a5042eed-9361-438a-a670-8b913eeac140\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " Nov 03 23:02:08.988351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988319 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08311447-fe46-43b3-aa87-86f1fa65136b-tls-certs\" (UniqueName: \"kubernetes.io/secret/08311447-fe46-43b3-aa87-86f1fa65136b-tls-certs\") pod \"08311447-fe46-43b3-aa87-86f1fa65136b\" (UID: \"08311447-fe46-43b3-aa87-86f1fa65136b\") " Nov 03 23:02:08.988589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988399 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-tmp-dir\") pod \"17e875ca-63d0-4219-af46-5247b25c98db\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " Nov 03 23:02:08.988589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988482 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-model-cache" pod "17e875ca-63d0-4219-af46-5247b25c98db" (UID: "17e875ca-63d0-4219-af46-5247b25c98db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.988589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988514 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-dshm\") pod \"17e875ca-63d0-4219-af46-5247b25c98db\" (UID: \"17e875ca-63d0-4219-af46-5247b25c98db\") " Nov 03 23:02:08.988746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988588 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-dshm\") pod \"a5042eed-9361-438a-a670-8b913eeac140\" (UID: \"a5042eed-9361-438a-a670-8b913eeac140\") " Nov 03 23:02:08.988746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.988674 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-home" pod "a5042eed-9361-438a-a670-8b913eeac140" (UID: "a5042eed-9361-438a-a670-8b913eeac140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989166 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989186 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-tmp-dir" pod "17e875ca-63d0-4219-af46-5247b25c98db" (UID: "17e875ca-63d0-4219-af46-5247b25c98db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989192 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989208 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-tmp-dir" pod "a5042eed-9361-438a-a670-8b913eeac140" (UID: "a5042eed-9361-438a-a670-8b913eeac140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989223 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989238 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989254 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989270 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:08.989359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.989284 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:08.990135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.990107 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e875ca-63d0-4219-af46-5247b25c98db-tls-certs" pod "17e875ca-63d0-4219-af46-5247b25c98db" (UID: "17e875ca-63d0-4219-af46-5247b25c98db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:08.991125 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.991090 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5042eed-9361-438a-a670-8b913eeac140-kube-api-access-t6d2v" pod "a5042eed-9361-438a-a670-8b913eeac140" (UID: "a5042eed-9361-438a-a670-8b913eeac140"). InnerVolumeSpecName "kube-api-access-t6d2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:08.991217 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.991191 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5042eed-9361-438a-a670-8b913eeac140-tls-certs" pod "a5042eed-9361-438a-a670-8b913eeac140" (UID: "a5042eed-9361-438a-a670-8b913eeac140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:08.991458 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.991425 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08311447-fe46-43b3-aa87-86f1fa65136b-kube-api-access-8cnk2" pod "08311447-fe46-43b3-aa87-86f1fa65136b" (UID: "08311447-fe46-43b3-aa87-86f1fa65136b"). InnerVolumeSpecName "kube-api-access-8cnk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:08.991624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.991597 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e875ca-63d0-4219-af46-5247b25c98db-kube-api-access-7rwh8" pod "17e875ca-63d0-4219-af46-5247b25c98db" (UID: "17e875ca-63d0-4219-af46-5247b25c98db"). InnerVolumeSpecName "kube-api-access-7rwh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:08.991696 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.991649 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-dshm" pod "08311447-fe46-43b3-aa87-86f1fa65136b" (UID: "08311447-fe46-43b3-aa87-86f1fa65136b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.992303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.992280 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-dshm" pod "17e875ca-63d0-4219-af46-5247b25c98db" (UID: "17e875ca-63d0-4219-af46-5247b25c98db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.992696 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.992673 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-dshm" pod "a5042eed-9361-438a-a670-8b913eeac140" (UID: "a5042eed-9361-438a-a670-8b913eeac140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:08.992846 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:08.992792 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08311447-fe46-43b3-aa87-86f1fa65136b-tls-certs" pod "08311447-fe46-43b3-aa87-86f1fa65136b" (UID: "08311447-fe46-43b3-aa87-86f1fa65136b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:09.090833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090784 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6d2v\" (UniqueName: \"kubernetes.io/projected/a5042eed-9361-438a-a670-8b913eeac140-kube-api-access-t6d2v\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.090833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090828 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e875ca-63d0-4219-af46-5247b25c98db-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.090833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090839 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5042eed-9361-438a-a670-8b913eeac140-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090848 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnk2\" (UniqueName: \"kubernetes.io/projected/08311447-fe46-43b3-aa87-86f1fa65136b-kube-api-access-8cnk2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090857 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwh8\" (UniqueName: \"kubernetes.io/projected/17e875ca-63d0-4219-af46-5247b25c98db-kube-api-access-7rwh8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090866 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08311447-fe46-43b3-aa87-86f1fa65136b-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090874 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090882 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08311447-fe46-43b3-aa87-86f1fa65136b-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090890 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090898 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e875ca-63d0-4219-af46-5247b25c98db-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.091148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.090906 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5042eed-9361-438a-a670-8b913eeac140-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.217631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.216792 5272 generic.go:356] "Generic (PLEG): container finished" podID="17e875ca-63d0-4219-af46-5247b25c98db" containerID="3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae" exitCode=0 Nov 03 23:02:09.217631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.216850 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" event={"ID":"17e875ca-63d0-4219-af46-5247b25c98db","Type":"ContainerDied","Data":"3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae"} Nov 03 23:02:09.217631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.216887 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" Nov 03 23:02:09.217631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.216900 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz" event={"ID":"17e875ca-63d0-4219-af46-5247b25c98db","Type":"ContainerDied","Data":"c894e516a8e816b25b526271b9db4ef6424690099a153f4997261d020e81fdc3"} Nov 03 23:02:09.217631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.216922 5272 scope.go:122] "RemoveContainer" containerID="3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae" Nov 03 23:02:09.220135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.220108 5272 generic.go:356] "Generic (PLEG): container finished" podID="08311447-fe46-43b3-aa87-86f1fa65136b" containerID="3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb" exitCode=0 Nov 03 23:02:09.220334 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.220315 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" Nov 03 23:02:09.220499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.220468 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" event={"ID":"08311447-fe46-43b3-aa87-86f1fa65136b","Type":"ContainerDied","Data":"3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb"} Nov 03 23:02:09.220681 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.220660 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499" event={"ID":"08311447-fe46-43b3-aa87-86f1fa65136b","Type":"ContainerDied","Data":"bbf1a7c8a3a6ffd2c965ed608540fe208bf9a3afa83f5ba6ff780efa1196c0af"} Nov 03 23:02:09.222202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.222178 5272 generic.go:356] "Generic (PLEG): container finished" podID="a5042eed-9361-438a-a670-8b913eeac140" containerID="a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804" exitCode=0 Nov 03 23:02:09.222452 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.222435 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" Nov 03 23:02:09.224604 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.223888 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" event={"ID":"a5042eed-9361-438a-a670-8b913eeac140","Type":"ContainerDied","Data":"a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804"} Nov 03 23:02:09.224604 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.223923 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb" event={"ID":"a5042eed-9361-438a-a670-8b913eeac140","Type":"ContainerDied","Data":"3c36e2288b91c5316fac3eaf7962383a80062c796c8ec791abd3fb3fa760ffd7"} Nov 03 23:02:09.233712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.233045 5272 scope.go:122] "RemoveContainer" containerID="3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae" Nov 03 23:02:09.233712 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:02:09.233374 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae\": container with ID starting with 3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae not found: ID does not exist" containerID="3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae" Nov 03 23:02:09.233712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.233411 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae"} err="failed to get container status \"3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae\": rpc error: code = NotFound desc = could not find container \"3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae\": container with ID starting with 3cf4f525d82abf4ebd1da25fab88447c965555217f1c701be227ba3cd8d5a6ae not found: ID does not exist" Nov 03 23:02:09.233712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.233433 5272 scope.go:122] "RemoveContainer" containerID="3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb" Nov 03 23:02:09.255576 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.255453 5272 scope.go:122] "RemoveContainer" containerID="3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb" Nov 03 23:02:09.264725 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:02:09.264302 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb\": container with ID starting with 3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb not found: ID does not exist" containerID="3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb" Nov 03 23:02:09.264725 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.264351 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb"} err="failed to get container status \"3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb\": rpc error: code = NotFound desc = could not find container \"3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb\": container with ID starting with 3ba5bde1cbc4c7d3b979d5379af8625495f9ea951b0306ad9eb019e0107a43bb not found: ID does not exist" Nov 03 23:02:09.264725 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.264418 5272 scope.go:122] "RemoveContainer" containerID="a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804" Nov 03 23:02:09.313093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.312824 5272 scope.go:122] "RemoveContainer" containerID="a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804" Nov 03 23:02:09.313395 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:02:09.313312 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804\": container with ID starting with a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804 not found: ID does not exist" containerID="a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804" Nov 03 23:02:09.313395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.313339 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804"} err="failed to get container status \"a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804\": rpc error: code = NotFound desc = could not find container \"a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804\": container with ID starting with a77552bbdb95450740fbe7555ab9596f0aee5c0503377133e44408bfe023e804 not found: ID does not exist" Nov 03 23:02:09.462322 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.462287 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:09.513258 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.513234 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:09.516709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.516689 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:09.520005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.519979 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499"] Nov 03 23:02:09.521049 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.521030 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:09.523140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.523117 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/premium-w2-ee74cb-kserve-6c84789dd8-fj499"] Nov 03 23:02:09.597540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597510 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-xks-ca\") pod \"8997735e-2b8b-4d54-87b4-d881a374c7e1\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " Nov 03 23:02:09.597654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597547 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-xks-ca\") pod \"d835ff5a-6984-4024-8234-f5e7e20bec9f\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " Nov 03 23:02:09.597654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597592 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8997735e-2b8b-4d54-87b4-d881a374c7e1-kube-api-access-px2hr\" (UniqueName: \"kubernetes.io/projected/8997735e-2b8b-4d54-87b4-d881a374c7e1-kube-api-access-px2hr\") pod \"8997735e-2b8b-4d54-87b4-d881a374c7e1\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " Nov 03 23:02:09.597781 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597657 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-openshift-ca\") pod \"d835ff5a-6984-4024-8234-f5e7e20bec9f\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " Nov 03 23:02:09.597781 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597723 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-xks-ca\") pod \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " Nov 03 23:02:09.597929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597779 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084a5e7e-1c6a-4342-94e1-0c7517486c1b-kube-api-access-6pfm8\" (UniqueName: \"kubernetes.io/projected/084a5e7e-1c6a-4342-94e1-0c7517486c1b-kube-api-access-6pfm8\") pod \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " Nov 03 23:02:09.597929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597850 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d835ff5a-6984-4024-8234-f5e7e20bec9f-kube-api-access-clbjl\" (UniqueName: \"kubernetes.io/projected/d835ff5a-6984-4024-8234-f5e7e20bec9f-kube-api-access-clbjl\") pod \"d835ff5a-6984-4024-8234-f5e7e20bec9f\" (UID: \"d835ff5a-6984-4024-8234-f5e7e20bec9f\") " Nov 03 23:02:09.597929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597881 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-openshift-ca\") pod \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " Nov 03 23:02:09.598104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597928 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-openshift-ca\") pod \"8997735e-2b8b-4d54-87b4-d881a374c7e1\" (UID: \"8997735e-2b8b-4d54-87b4-d881a374c7e1\") " Nov 03 23:02:09.598104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.597986 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-kube-api-access-ht92t\") pod \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " Nov 03 23:02:09.598104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598074 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-openshift-ca" pod "d835ff5a-6984-4024-8234-f5e7e20bec9f" (UID: "d835ff5a-6984-4024-8234-f5e7e20bec9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:09.598259 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598118 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-openshift-ca\") pod \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\" (UID: \"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958\") " Nov 03 23:02:09.598259 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598153 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-xks-ca\") pod \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\" (UID: \"084a5e7e-1c6a-4342-94e1-0c7517486c1b\") " Nov 03 23:02:09.598853 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598472 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-openshift-ca" pod "084a5e7e-1c6a-4342-94e1-0c7517486c1b" (UID: "084a5e7e-1c6a-4342-94e1-0c7517486c1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:09.598853 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598712 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-openshift-ca" pod "2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" (UID: "2f6db47f-e8c1-4ed8-8a31-bbc28e01e958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:09.598853 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598735 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.598853 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598757 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.599118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.598951 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-openshift-ca" pod "8997735e-2b8b-4d54-87b4-d881a374c7e1" (UID: "8997735e-2b8b-4d54-87b4-d881a374c7e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:09.600313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.600245 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-xks-ca" pod "d835ff5a-6984-4024-8234-f5e7e20bec9f" (UID: "d835ff5a-6984-4024-8234-f5e7e20bec9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:09.600313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.600251 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-xks-ca" pod "8997735e-2b8b-4d54-87b4-d881a374c7e1" (UID: "8997735e-2b8b-4d54-87b4-d881a374c7e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:09.600735 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.600708 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8997735e-2b8b-4d54-87b4-d881a374c7e1-kube-api-access-px2hr" pod "8997735e-2b8b-4d54-87b4-d881a374c7e1" (UID: "8997735e-2b8b-4d54-87b4-d881a374c7e1"). InnerVolumeSpecName "kube-api-access-px2hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:09.601500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.600960 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-kube-api-access-ht92t" pod "2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" (UID: "2f6db47f-e8c1-4ed8-8a31-bbc28e01e958"). InnerVolumeSpecName "kube-api-access-ht92t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:09.601500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.600978 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-xks-ca" pod "2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" (UID: "2f6db47f-e8c1-4ed8-8a31-bbc28e01e958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:09.601500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.601077 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084a5e7e-1c6a-4342-94e1-0c7517486c1b-kube-api-access-6pfm8" pod "084a5e7e-1c6a-4342-94e1-0c7517486c1b" (UID: "084a5e7e-1c6a-4342-94e1-0c7517486c1b"). InnerVolumeSpecName "kube-api-access-6pfm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:09.601692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.601571 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d835ff5a-6984-4024-8234-f5e7e20bec9f-kube-api-access-clbjl" pod "d835ff5a-6984-4024-8234-f5e7e20bec9f" (UID: "d835ff5a-6984-4024-8234-f5e7e20bec9f"). InnerVolumeSpecName "kube-api-access-clbjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:09.601766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.601740 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-xks-ca" pod "084a5e7e-1c6a-4342-94e1-0c7517486c1b" (UID: "084a5e7e-1c6a-4342-94e1-0c7517486c1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:09.625278 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.625256 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb"] Nov 03 23:02:09.630626 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.630602 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-w2-ee74cb-kserve-64f486db99-9zlvb"] Nov 03 23:02:09.699695 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699670 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699695 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699692 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a5e7e-1c6a-4342-94e1-0c7517486c1b-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699703 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699711 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d835ff5a-6984-4024-8234-f5e7e20bec9f-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699721 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px2hr\" (UniqueName: \"kubernetes.io/projected/8997735e-2b8b-4d54-87b4-d881a374c7e1-kube-api-access-px2hr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699730 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699738 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfm8\" (UniqueName: \"kubernetes.io/projected/084a5e7e-1c6a-4342-94e1-0c7517486c1b-kube-api-access-6pfm8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699747 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbjl\" (UniqueName: \"kubernetes.io/projected/d835ff5a-6984-4024-8234-f5e7e20bec9f-kube-api-access-clbjl\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699756 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8997735e-2b8b-4d54-87b4-d881a374c7e1-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.699873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.699766 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958-kube-api-access-ht92t\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:09.732516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.732490 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz"] Nov 03 23:02:09.737335 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:09.737312 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ee74cb/distinct-2-w2-ee74cb-kserve-5cdcbf4fb4-r9wtz"] Nov 03 23:02:10.231468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.231381 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" event={"ID":"2f6db47f-e8c1-4ed8-8a31-bbc28e01e958","Type":"ContainerDied","Data":"23fa4e401c015880b18b7f9b15f704259732e7fea4d971b08a5b90a8ca1f18be"} Nov 03 23:02:10.231468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.231461 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23fa4e401c015880b18b7f9b15f704259732e7fea4d971b08a5b90a8ca1f18be" Nov 03 23:02:10.231959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.231395 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls" Nov 03 23:02:10.232765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.232739 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" Nov 03 23:02:10.232765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.232758 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb" event={"ID":"084a5e7e-1c6a-4342-94e1-0c7517486c1b","Type":"ContainerDied","Data":"9c15f39b4bfb7ba24840dee3a07f65f93251fc599b18377886a543fa88527459"} Nov 03 23:02:10.232949 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.232797 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c15f39b4bfb7ba24840dee3a07f65f93251fc599b18377886a543fa88527459" Nov 03 23:02:10.235422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.235370 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" Nov 03 23:02:10.235538 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.235381 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76" event={"ID":"d835ff5a-6984-4024-8234-f5e7e20bec9f","Type":"ContainerDied","Data":"b3ebf876f679bb3344807b7bce58fe7e6b0cbebcc7456794c6b1e641bc51260a"} Nov 03 23:02:10.235599 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.235541 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3ebf876f679bb3344807b7bce58fe7e6b0cbebcc7456794c6b1e641bc51260a" Nov 03 23:02:10.239203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.239177 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" event={"ID":"8997735e-2b8b-4d54-87b4-d881a374c7e1","Type":"ContainerDied","Data":"474bd32feee19c883abca7d20270a1833cc24a81f5b06720925bcbd2f60dc57d"} Nov 03 23:02:10.239306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.239213 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474bd32feee19c883abca7d20270a1833cc24a81f5b06720925bcbd2f60dc57d" Nov 03 23:02:10.239306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:10.239224 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq" Nov 03 23:02:11.494591 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:11.494555 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" path="/var/lib/kubelet/pods/08311447-fe46-43b3-aa87-86f1fa65136b/volumes" Nov 03 23:02:11.495015 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:11.494986 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e875ca-63d0-4219-af46-5247b25c98db" path="/var/lib/kubelet/pods/17e875ca-63d0-4219-af46-5247b25c98db/volumes" Nov 03 23:02:11.495378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:11.495364 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5042eed-9361-438a-a670-8b913eeac140" path="/var/lib/kubelet/pods/a5042eed-9361-438a-a670-8b913eeac140/volumes" Nov 03 23:02:12.593256 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593222 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5"] Nov 03 23:02:12.593845 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593782 5272 cpu_manager.go:414] "containerMap: removing container" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" Nov 03 23:02:12.593845 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593812 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" Nov 03 23:02:12.593845 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593837 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d835ff5a-6984-4024-8234-f5e7e20bec9f" containerName="revoke-keys" Nov 03 23:02:12.593845 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593846 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d835ff5a-6984-4024-8234-f5e7e20bec9f" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593870 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8997735e-2b8b-4d54-87b4-d881a374c7e1" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593878 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8997735e-2b8b-4d54-87b4-d881a374c7e1" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593896 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593903 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593929 5272 cpu_manager.go:414] "containerMap: removing container" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593934 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593945 5272 cpu_manager.go:414] "containerMap: removing container" podUID="084a5e7e-1c6a-4342-94e1-0c7517486c1b" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593949 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084a5e7e-1c6a-4342-94e1-0c7517486c1b" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593956 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.593961 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.594083 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8997735e-2b8b-4d54-87b4-d881a374c7e1" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.594096 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.594106 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e875ca-63d0-4219-af46-5247b25c98db" containerName="main" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.594113 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="084a5e7e-1c6a-4342-94e1-0c7517486c1b" containerName="revoke-keys" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.594120 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5042eed-9361-438a-a670-8b913eeac140" containerName="main" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.594129 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="08311447-fe46-43b3-aa87-86f1fa65136b" containerName="main" Nov 03 23:02:12.594180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.594136 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d835ff5a-6984-4024-8234-f5e7e20bec9f" containerName="revoke-keys" Nov 03 23:02:12.599089 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.599070 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.677778 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.677740 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5"] Nov 03 23:02:12.695038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.695009 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b4383\"/\"default-dockercfg-f9mdc\"" Nov 03 23:02:12.695038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.695032 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b4383\"/\"openshift-service-ca.crt\"" Nov 03 23:02:12.695223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.695049 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b4383\"/\"kube-root-ca.crt\"" Nov 03 23:02:12.695223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.695094 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b4383\"/\"test-model-9b4383-kserve-self-signed-certs\"" Nov 03 23:02:12.787374 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.787324 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-dshm\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.787549 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.787419 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-tmp-dir\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.787549 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.787453 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrwq\" (UniqueName: \"kubernetes.io/projected/7677b675-ffb7-4481-beec-5a7b78fae29b-kube-api-access-whrwq\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.787549 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.787496 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-home\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.787696 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.787568 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-model-cache\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.787751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.787725 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7677b675-ffb7-4481-beec-5a7b78fae29b-tls-certs\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.850929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.850835 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" podUID="fe803d4b-d693-4f97-9f1c-c495aaea0276" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:02:12.850929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.850849 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" podUID="299f09cd-663e-4790-8f75-8f1f2698571c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:02:12.888730 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.888690 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7677b675-ffb7-4481-beec-5a7b78fae29b-tls-certs\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.888959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.888749 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-dshm\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.888959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.888779 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-tmp-dir\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.888959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.888796 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrwq\" (UniqueName: \"kubernetes.io/projected/7677b675-ffb7-4481-beec-5a7b78fae29b-kube-api-access-whrwq\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.888959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.888880 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-home\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.888959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.888950 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-model-cache\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.889320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.889300 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-tmp-dir\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.889366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.889317 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-model-cache\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.889404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.889371 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-home\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.891136 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.891108 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-dshm\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.891433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.891412 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7677b675-ffb7-4481-beec-5a7b78fae29b-tls-certs\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:12.946237 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:12.946200 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrwq\" (UniqueName: \"kubernetes.io/projected/7677b675-ffb7-4481-beec-5a7b78fae29b-kube-api-access-whrwq\") pod \"test-model-9b4383-kserve-796d57944f-k67d5\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:13.217878 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:13.214024 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:13.656617 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:13.656582 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5"] Nov 03 23:02:13.662041 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:13.662007 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7677b675_ffb7_4481_beec_5a7b78fae29b.slice/crio-30bdaac8d43dbe52eba2b8fe4e2b3f5751dd119fdb9d7e6b22950d04c4028337 WatchSource:0}: Error finding container 30bdaac8d43dbe52eba2b8fe4e2b3f5751dd119fdb9d7e6b22950d04c4028337: Status 404 returned error can't find the container with id 30bdaac8d43dbe52eba2b8fe4e2b3f5751dd119fdb9d7e6b22950d04c4028337 Nov 03 23:02:14.311426 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:14.311385 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" event={"ID":"7677b675-ffb7-4481-beec-5a7b78fae29b","Type":"ContainerStarted","Data":"c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b"} Nov 03 23:02:14.311607 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:14.311438 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" event={"ID":"7677b675-ffb7-4481-beec-5a7b78fae29b","Type":"ContainerStarted","Data":"30bdaac8d43dbe52eba2b8fe4e2b3f5751dd119fdb9d7e6b22950d04c4028337"} Nov 03 23:02:14.424596 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:14.424543 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" podStartSLOduration=2.424528387 podStartE2EDuration="2.424528387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:14.422036716 +0000 UTC m=+2315.044345025" watchObservedRunningTime="2026-11-03 23:02:14.424528387 +0000 UTC m=+2315.046836695" Nov 03 23:02:15.221641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:15.221610 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:22.850910 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:22.850866 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" podUID="fe803d4b-d693-4f97-9f1c-c495aaea0276" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:02:22.851424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:22.850928 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" podUID="299f09cd-663e-4790-8f75-8f1f2698571c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:02:23.226875 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:23.221693 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:23.243673 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:23.243641 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:23.352718 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:23.352681 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk"] Nov 03 23:02:23.356327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:23.356303 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-961054-f9tzk"] Nov 03 23:02:23.468290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:23.468259 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:02:23.493432 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:23.493356 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b682ae1-5f26-4d7d-930b-451d818bc172" path="/var/lib/kubelet/pods/3b682ae1-5f26-4d7d-930b-451d818bc172/volumes" Nov 03 23:02:24.500612 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:02:24.500575 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:02:26.484623 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.484594 5272 generic.go:356] "Generic (PLEG): container finished" podID="299f09cd-663e-4790-8f75-8f1f2698571c" containerID="559668611da7931d5e7aa5ccd1085cae651bd1030a0bee233be184a56b399d21" exitCode=0 Nov 03 23:02:26.485042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.484672 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" event={"ID":"299f09cd-663e-4790-8f75-8f1f2698571c","Type":"ContainerDied","Data":"559668611da7931d5e7aa5ccd1085cae651bd1030a0bee233be184a56b399d21"} Nov 03 23:02:26.486575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.486552 5272 generic.go:356] "Generic (PLEG): container finished" podID="fe803d4b-d693-4f97-9f1c-c495aaea0276" containerID="dfaa09cfebcb16909b13d36558bce92d205ac645d57ae3f608d079e861d24b88" exitCode=0 Nov 03 23:02:26.486713 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.486701 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" event={"ID":"fe803d4b-d693-4f97-9f1c-c495aaea0276","Type":"ContainerDied","Data":"dfaa09cfebcb16909b13d36558bce92d205ac645d57ae3f608d079e861d24b88"} Nov 03 23:02:26.521760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.521698 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 23:02:26.573134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.573105 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe803d4b-d693-4f97-9f1c-c495aaea0276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe803d4b-d693-4f97-9f1c-c495aaea0276-plugins-config-volume\") pod \"fe803d4b-d693-4f97-9f1c-c495aaea0276\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " Nov 03 23:02:26.573290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.573238 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe803d4b-d693-4f97-9f1c-c495aaea0276-kube-api-access-ck8hv\" (UniqueName: \"kubernetes.io/projected/fe803d4b-d693-4f97-9f1c-c495aaea0276-kube-api-access-ck8hv\") pod \"fe803d4b-d693-4f97-9f1c-c495aaea0276\" (UID: \"fe803d4b-d693-4f97-9f1c-c495aaea0276\") " Nov 03 23:02:26.573685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.573469 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe803d4b-d693-4f97-9f1c-c495aaea0276-plugins-config-volume" pod "fe803d4b-d693-4f97-9f1c-c495aaea0276" (UID: "fe803d4b-d693-4f97-9f1c-c495aaea0276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:26.573778 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.573684 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe803d4b-d693-4f97-9f1c-c495aaea0276-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:26.576736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.576706 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe803d4b-d693-4f97-9f1c-c495aaea0276-kube-api-access-ck8hv" pod "fe803d4b-d693-4f97-9f1c-c495aaea0276" (UID: "fe803d4b-d693-4f97-9f1c-c495aaea0276"). InnerVolumeSpecName "kube-api-access-ck8hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:26.602708 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.602689 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 23:02:26.675105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.675074 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299f09cd-663e-4790-8f75-8f1f2698571c-kube-api-access-9mzqj\" (UniqueName: \"kubernetes.io/projected/299f09cd-663e-4790-8f75-8f1f2698571c-kube-api-access-9mzqj\") pod \"299f09cd-663e-4790-8f75-8f1f2698571c\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " Nov 03 23:02:26.675303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.675285 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299f09cd-663e-4790-8f75-8f1f2698571c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299f09cd-663e-4790-8f75-8f1f2698571c-plugins-config-volume\") pod \"299f09cd-663e-4790-8f75-8f1f2698571c\" (UID: \"299f09cd-663e-4790-8f75-8f1f2698571c\") " Nov 03 23:02:26.675629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.675606 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299f09cd-663e-4790-8f75-8f1f2698571c-plugins-config-volume" pod "299f09cd-663e-4790-8f75-8f1f2698571c" (UID: "299f09cd-663e-4790-8f75-8f1f2698571c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:26.675781 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.675760 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299f09cd-663e-4790-8f75-8f1f2698571c-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:26.675896 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.675788 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck8hv\" (UniqueName: \"kubernetes.io/projected/fe803d4b-d693-4f97-9f1c-c495aaea0276-kube-api-access-ck8hv\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:26.677234 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.677212 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299f09cd-663e-4790-8f75-8f1f2698571c-kube-api-access-9mzqj" pod "299f09cd-663e-4790-8f75-8f1f2698571c" (UID: "299f09cd-663e-4790-8f75-8f1f2698571c"). InnerVolumeSpecName "kube-api-access-9mzqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:26.777400 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:26.777332 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzqj\" (UniqueName: \"kubernetes.io/projected/299f09cd-663e-4790-8f75-8f1f2698571c-kube-api-access-9mzqj\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:27.496348 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.496316 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" Nov 03 23:02:27.496831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.496313 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5" event={"ID":"299f09cd-663e-4790-8f75-8f1f2698571c","Type":"ContainerDied","Data":"d967d27aaebfbbc919d5f982c96b47d75cddc244344c9944f3399e2ce2e0e115"} Nov 03 23:02:27.496831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.496488 5272 scope.go:122] "RemoveContainer" containerID="559668611da7931d5e7aa5ccd1085cae651bd1030a0bee233be184a56b399d21" Nov 03 23:02:27.498027 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.498000 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" event={"ID":"fe803d4b-d693-4f97-9f1c-c495aaea0276","Type":"ContainerDied","Data":"90d190aee65b3bf3f102dad9daa0e67f65802aa17532638a643cce54793920f3"} Nov 03 23:02:27.498130 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.498035 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss" Nov 03 23:02:27.507434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.507339 5272 scope.go:122] "RemoveContainer" containerID="dfaa09cfebcb16909b13d36558bce92d205ac645d57ae3f608d079e861d24b88" Nov 03 23:02:27.651284 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.651253 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss"] Nov 03 23:02:27.697744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.697713 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee74cb-6479cc6f4b-5qpss"] Nov 03 23:02:27.756669 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.756641 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5"] Nov 03 23:02:27.760788 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:27.760766 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ee74cb-5786ddf6ff-lb7f5"] Nov 03 23:02:29.492946 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:29.492904 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299f09cd-663e-4790-8f75-8f1f2698571c" path="/var/lib/kubelet/pods/299f09cd-663e-4790-8f75-8f1f2698571c/volumes" Nov 03 23:02:29.493504 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:29.493482 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe803d4b-d693-4f97-9f1c-c495aaea0276" path="/var/lib/kubelet/pods/fe803d4b-d693-4f97-9f1c-c495aaea0276/volumes" Nov 03 23:02:32.916756 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:32.916721 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:02:43.298135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:43.298016 5272 scope.go:122] "RemoveContainer" containerID="e73b417521c6cc5fc2cd782db954547983639834e4bb5338a46acf446260fd5f" Nov 03 23:02:43.578996 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:43.578912 5272 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:02:43.578996 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:43.578976 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:02:44.441854 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.441728 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442"] Nov 03 23:02:44.487925 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.487893 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a03fee-7r442"] Nov 03 23:02:44.706455 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.706379 5272 generic.go:356] "Generic (PLEG): container finished" podID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerID="0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972" exitCode=0 Nov 03 23:02:44.706615 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.706457 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" event={"ID":"10f3ea73-2aeb-40ed-990a-31a494afd1ed","Type":"ContainerDied","Data":"0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972"} Nov 03 23:02:44.706878 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.706863 5272 scope.go:122] "RemoveContainer" containerID="0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972" Nov 03 23:02:44.708224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.708188 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:44.708344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.708274 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:44.708344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:44.708324 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:45.494239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:45.494208 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1811472-e347-410a-bf73-7794f8ec4267" path="/var/lib/kubelet/pods/d1811472-e347-410a-bf73-7794f8ec4267/volumes" Nov 03 23:02:45.717114 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:45.717072 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" event={"ID":"10f3ea73-2aeb-40ed-990a-31a494afd1ed","Type":"ContainerStarted","Data":"0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407"} Nov 03 23:02:46.094166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:46.094130 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:02:46.094381 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:46.094361 5272 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 23:02:46.094431 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:46.094410 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 23:02:47.093446 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:47.093413 5272 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 23:02:47.093890 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:47.093465 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 23:02:48.093442 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:48.093404 5272 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 23:02:48.093623 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:48.093462 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 23:02:49.094015 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:49.093979 5272 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 23:02:49.094426 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:49.094031 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 23:02:50.093495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:50.093360 5272 patch_prober.go:29] interesting pod/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 03 23:02:50.093495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:50.093417 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 03 23:02:51.097636 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:51.097601 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:02:51.098128 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:51.098104 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:02:51.098857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:51.098836 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:02:51.673557 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:51.673519 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj"] Nov 03 23:02:51.673876 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:51.673827 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" containerID="cri-o://36057d33e274c96702e6bc1046f8a14e6e4986145f6e33d9bb7944017dba842d" gracePeriod=30 Nov 03 23:02:52.673148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.673118 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s"] Nov 03 23:02:52.673744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.673719 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fe803d4b-d693-4f97-9f1c-c495aaea0276" containerName="payload-processing" Nov 03 23:02:52.673744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.673741 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe803d4b-d693-4f97-9f1c-c495aaea0276" containerName="payload-processing" Nov 03 23:02:52.673970 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.673822 5272 cpu_manager.go:414] "containerMap: removing container" podUID="299f09cd-663e-4790-8f75-8f1f2698571c" containerName="payload-pre-processing" Nov 03 23:02:52.673970 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.673832 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299f09cd-663e-4790-8f75-8f1f2698571c" containerName="payload-pre-processing" Nov 03 23:02:52.674061 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.674010 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="299f09cd-663e-4790-8f75-8f1f2698571c" containerName="payload-pre-processing" Nov 03 23:02:52.674061 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.674028 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe803d4b-d693-4f97-9f1c-c495aaea0276" containerName="payload-processing" Nov 03 23:02:52.676835 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.676792 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.727251 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.727230 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:02:52.731186 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.731157 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s"] Nov 03 23:02:52.738040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.738012 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r58hm\" (UniqueName: \"kubernetes.io/projected/975e5eed-7cd0-4ce1-8538-539ac45e8433-kube-api-access-r58hm\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.738230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.738210 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.738302 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.738241 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.773907 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.773876 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:52.774025 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.773923 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:02:52.812125 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.812101 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:02:52.839228 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.839203 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r58hm\" (UniqueName: \"kubernetes.io/projected/975e5eed-7cd0-4ce1-8538-539ac45e8433-kube-api-access-r58hm\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.839403 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.839370 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.839529 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.839410 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.840107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.840084 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.841612 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.841591 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.898306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.898273 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r58hm\" (UniqueName: \"kubernetes.io/projected/975e5eed-7cd0-4ce1-8538-539ac45e8433-kube-api-access-r58hm\") pod \"maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:52.989263 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:52.989192 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:53.370284 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:53.370248 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s"] Nov 03 23:02:53.376249 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:53.376222 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975e5eed_7cd0_4ce1_8538_539ac45e8433.slice/crio-3942e3151349e167e0426bcf0866533a96e5b6d2a6e834b5503351e818ad1b34 WatchSource:0}: Error finding container 3942e3151349e167e0426bcf0866533a96e5b6d2a6e834b5503351e818ad1b34: Status 404 returned error can't find the container with id 3942e3151349e167e0426bcf0866533a96e5b6d2a6e834b5503351e818ad1b34 Nov 03 23:02:53.839980 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:53.839941 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" event={"ID":"975e5eed-7cd0-4ce1-8538-539ac45e8433","Type":"ContainerStarted","Data":"4098c4826f12837d0ef4773a7a195328f4d7016453023af16047cf978d36be6c"} Nov 03 23:02:53.840366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:53.840009 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" event={"ID":"975e5eed-7cd0-4ce1-8538-539ac45e8433","Type":"ContainerStarted","Data":"3942e3151349e167e0426bcf0866533a96e5b6d2a6e834b5503351e818ad1b34"} Nov 03 23:02:53.949356 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:53.949259 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" podStartSLOduration=1.949244704 podStartE2EDuration="1.949244704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:53.947725223 +0000 UTC m=+2354.570033532" watchObservedRunningTime="2026-11-03 23:02:53.949244704 +0000 UTC m=+2354.571553016" Nov 03 23:02:54.500559 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:02:54.500476 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:02:54.848456 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:54.848417 5272 generic.go:356] "Generic (PLEG): container finished" podID="975e5eed-7cd0-4ce1-8538-539ac45e8433" containerID="4098c4826f12837d0ef4773a7a195328f4d7016453023af16047cf978d36be6c" exitCode=0 Nov 03 23:02:54.848877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:54.848495 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" event={"ID":"975e5eed-7cd0-4ce1-8538-539ac45e8433","Type":"ContainerDied","Data":"4098c4826f12837d0ef4773a7a195328f4d7016453023af16047cf978d36be6c"} Nov 03 23:02:56.008726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.008700 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:56.076885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.076847 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-openshift-ca\") pod \"975e5eed-7cd0-4ce1-8538-539ac45e8433\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " Nov 03 23:02:56.077037 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.076908 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975e5eed-7cd0-4ce1-8538-539ac45e8433-kube-api-access-r58hm\" (UniqueName: \"kubernetes.io/projected/975e5eed-7cd0-4ce1-8538-539ac45e8433-kube-api-access-r58hm\") pod \"975e5eed-7cd0-4ce1-8538-539ac45e8433\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " Nov 03 23:02:56.077037 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.077015 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-xks-ca\") pod \"975e5eed-7cd0-4ce1-8538-539ac45e8433\" (UID: \"975e5eed-7cd0-4ce1-8538-539ac45e8433\") " Nov 03 23:02:56.077230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.077202 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-openshift-ca" pod "975e5eed-7cd0-4ce1-8538-539ac45e8433" (UID: "975e5eed-7cd0-4ce1-8538-539ac45e8433"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:56.077438 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.077418 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.079064 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.079035 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-xks-ca" pod "975e5eed-7cd0-4ce1-8538-539ac45e8433" (UID: "975e5eed-7cd0-4ce1-8538-539ac45e8433"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:56.079208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.079187 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975e5eed-7cd0-4ce1-8538-539ac45e8433-kube-api-access-r58hm" pod "975e5eed-7cd0-4ce1-8538-539ac45e8433" (UID: "975e5eed-7cd0-4ce1-8538-539ac45e8433"). InnerVolumeSpecName "kube-api-access-r58hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:56.179044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.178957 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975e5eed-7cd0-4ce1-8538-539ac45e8433-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.179044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.178988 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r58hm\" (UniqueName: \"kubernetes.io/projected/975e5eed-7cd0-4ce1-8538-539ac45e8433-kube-api-access-r58hm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.865936 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.865906 5272 generic.go:356] "Generic (PLEG): container finished" podID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerID="36057d33e274c96702e6bc1046f8a14e6e4986145f6e33d9bb7944017dba842d" exitCode=0 Nov 03 23:02:56.866105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.865984 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" event={"ID":"1ed69d8c-da98-4c76-8d68-b475ebb41024","Type":"ContainerDied","Data":"36057d33e274c96702e6bc1046f8a14e6e4986145f6e33d9bb7944017dba842d"} Nov 03 23:02:56.866105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.866056 5272 scope.go:122] "RemoveContainer" containerID="3fe0c16cc46fb6b435a46726669df7003222fc6b2f432d77b83a33f2d4078501" Nov 03 23:02:56.867607 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.867587 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" Nov 03 23:02:56.867712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.867604 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s" event={"ID":"975e5eed-7cd0-4ce1-8538-539ac45e8433","Type":"ContainerDied","Data":"3942e3151349e167e0426bcf0866533a96e5b6d2a6e834b5503351e818ad1b34"} Nov 03 23:02:56.867712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.867646 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3942e3151349e167e0426bcf0866533a96e5b6d2a6e834b5503351e818ad1b34" Nov 03 23:02:56.923032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.923008 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 23:02:56.988460 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988400 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-token\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.988608 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988490 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-socket\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.988608 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988570 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-envoy\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.988730 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988627 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed69d8c-da98-4c76-8d68-b475ebb41024-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed69d8c-da98-4c76-8d68-b475ebb41024-istiod-ca-cert\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.988730 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988708 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-kube-api-access-sxjpc\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-kube-api-access-sxjpc\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.988873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988745 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-podinfo\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.988873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988794 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-credential-socket\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.988873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988847 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-certs\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.989038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.988899 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-data\") pod \"1ed69d8c-da98-4c76-8d68-b475ebb41024\" (UID: \"1ed69d8c-da98-4c76-8d68-b475ebb41024\") " Nov 03 23:02:56.989038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989002 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed69d8c-da98-4c76-8d68-b475ebb41024-istiod-ca-cert" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:56.989147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989040 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-credential-socket" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:56.989147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989077 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-socket" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:56.989300 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989277 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-certs" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:56.989300 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989291 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-data" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:56.989536 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989512 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed69d8c-da98-4c76-8d68-b475ebb41024-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.989631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989541 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.989631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989556 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.989631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989571 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.989631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.989586 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:56.990939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.990855 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-podinfo" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:02:56.991167 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.991140 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-envoy" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:56.991364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.991271 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-token" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:56.991442 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:56.991337 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-kube-api-access-sxjpc" pod "1ed69d8c-da98-4c76-8d68-b475ebb41024" (UID: "1ed69d8c-da98-4c76-8d68-b475ebb41024"). InnerVolumeSpecName "kube-api-access-sxjpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:57.090308 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.090274 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjpc\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-kube-api-access-sxjpc\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:57.090308 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.090304 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:57.090308 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.090315 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:57.090923 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.090324 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed69d8c-da98-4c76-8d68-b475ebb41024-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:57.143326 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.143292 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp"] Nov 03 23:02:57.163867 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.163841 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh"] Nov 03 23:02:57.875193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.875160 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" Nov 03 23:02:57.875376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.875197 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj" event={"ID":"1ed69d8c-da98-4c76-8d68-b475ebb41024","Type":"ContainerDied","Data":"91d13ca83274132e5fdce7ba64dc860331eb0b38c77b10bad874dd44cdcf3505"} Nov 03 23:02:57.875376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:57.875262 5272 scope.go:122] "RemoveContainer" containerID="36057d33e274c96702e6bc1046f8a14e6e4986145f6e33d9bb7944017dba842d" Nov 03 23:02:58.031685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.031650 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj"] Nov 03 23:02:58.078296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.078267 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a03fee-openshift-default-5f8f5cbff7-pzmcj"] Nov 03 23:02:58.129747 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.129679 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5"] Nov 03 23:02:58.130226 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130211 5272 cpu_manager.go:414] "containerMap: removing container" podUID="975e5eed-7cd0-4ce1-8538-539ac45e8433" containerName="revoke-keys" Nov 03 23:02:58.130286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130226 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975e5eed-7cd0-4ce1-8538-539ac45e8433" containerName="revoke-keys" Nov 03 23:02:58.130286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130249 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" Nov 03 23:02:58.130286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130254 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" Nov 03 23:02:58.130286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130269 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" Nov 03 23:02:58.130286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130274 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" Nov 03 23:02:58.130455 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130365 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" Nov 03 23:02:58.130455 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.130379 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="975e5eed-7cd0-4ce1-8538-539ac45e8433" containerName="revoke-keys" Nov 03 23:02:58.134418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.134399 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.192970 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.192937 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5"] Nov 03 23:02:58.203242 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203210 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce868080-9616-4fec-9184-9363ec40badd-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203397 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203252 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce868080-9616-4fec-9184-9363ec40badd-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203463 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203397 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203456 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203502 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203589 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203690 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203667 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7rw\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-kube-api-access-rb7rw\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203846 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.203919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.203883 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.304873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.304844 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce868080-9616-4fec-9184-9363ec40badd-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.304873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.304878 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce868080-9616-4fec-9184-9363ec40badd-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305012 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305060 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305088 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305137 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305176 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7rw\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-kube-api-access-rb7rw\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305390 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305296 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305390 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305322 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305470 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305572 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305497 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-certs\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305572 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305563 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce868080-9616-4fec-9184-9363ec40badd-istiod-ca-cert\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305717 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305694 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-data\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.305777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.305748 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-credential-socket\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.307332 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.307306 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-envoy\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.307448 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.307402 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce868080-9616-4fec-9184-9363ec40badd-istio-podinfo\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.361979 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.361950 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-istio-token\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.362140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.362123 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7rw\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-kube-api-access-rb7rw\") pod \"e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.447659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.447602 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:02:58.654772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.654742 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77"] Nov 03 23:02:58.655558 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.655535 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" containerName="istio-proxy" Nov 03 23:02:58.659086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.659065 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.708171 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.708120 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ee74cb-openshift-default-dockercfg-zszk2\"" Nov 03 23:02:58.714366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714344 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.714494 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714377 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.714494 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714418 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.714663 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714634 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.714736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714719 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.714833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714817 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.714879 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714855 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.714921 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.714889 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.715105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.715082 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8d8\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-kube-api-access-bh8d8\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.717221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.717199 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77"] Nov 03 23:02:58.816403 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816372 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816429 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816477 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816498 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816518 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816779 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816566 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8d8\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-kube-api-access-bh8d8\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816779 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816596 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816779 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816619 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.816779 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816679 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.817108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816975 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.817108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.816997 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-certs\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.817229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.817155 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-credential-socket\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.817297 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.817261 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-data\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.817442 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.817423 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istiod-ca-cert\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.818794 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.818773 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-envoy\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.818904 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.818849 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-podinfo\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.873936 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.873909 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-token\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.874073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.874036 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8d8\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-kube-api-access-bh8d8\") pod \"e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:58.887107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.887077 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5"] Nov 03 23:02:58.893157 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:58.892477 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce868080_9616_4fec_9184_9363ec40badd.slice/crio-771b249e47984e753c5e67616f59e36735bec15b1bf8ed99a00935d65686b1d3 WatchSource:0}: Error finding container 771b249e47984e753c5e67616f59e36735bec15b1bf8ed99a00935d65686b1d3: Status 404 returned error can't find the container with id 771b249e47984e753c5e67616f59e36735bec15b1bf8ed99a00935d65686b1d3 Nov 03 23:02:58.895218 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.895181 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:58.895303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.895258 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:58.895303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.895289 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:58.972328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:58.972270 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:02:59.413527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.413489 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77"] Nov 03 23:02:59.418666 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:02:59.418639 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f505e0_4538_4cd8_b040_7910b2f8d54e.slice/crio-21a960b00ec8adca06fc8314e370018ff55289f1dee4502c863e376e695666b0 WatchSource:0}: Error finding container 21a960b00ec8adca06fc8314e370018ff55289f1dee4502c863e376e695666b0: Status 404 returned error can't find the container with id 21a960b00ec8adca06fc8314e370018ff55289f1dee4502c863e376e695666b0 Nov 03 23:02:59.421240 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.421206 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:59.421312 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.421275 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:59.421354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.421320 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:02:59.494342 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.494316 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed69d8c-da98-4c76-8d68-b475ebb41024" path="/var/lib/kubelet/pods/1ed69d8c-da98-4c76-8d68-b475ebb41024/volumes" Nov 03 23:02:59.891720 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.891664 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" event={"ID":"a5f505e0-4538-4cd8-b040-7910b2f8d54e","Type":"ContainerStarted","Data":"4c96b3a826da721040a6942131fd7d1dd5eb60e16bf10e38cb6939fae4a714f0"} Nov 03 23:02:59.891945 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.891773 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" event={"ID":"a5f505e0-4538-4cd8-b040-7910b2f8d54e","Type":"ContainerStarted","Data":"21a960b00ec8adca06fc8314e370018ff55289f1dee4502c863e376e695666b0"} Nov 03 23:02:59.893715 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.893678 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" event={"ID":"ce868080-9616-4fec-9184-9363ec40badd","Type":"ContainerStarted","Data":"d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2"} Nov 03 23:02:59.893877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.893729 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" event={"ID":"ce868080-9616-4fec-9184-9363ec40badd","Type":"ContainerStarted","Data":"771b249e47984e753c5e67616f59e36735bec15b1bf8ed99a00935d65686b1d3"} Nov 03 23:02:59.973864 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:02:59.973724 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:03:00.009515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:00.009450 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podStartSLOduration=2.009430856 podStartE2EDuration="2.009430856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:00.004254826 +0000 UTC m=+2360.626563134" watchObservedRunningTime="2026-11-03 23:03:00.009430856 +0000 UTC m=+2360.631739166" Nov 03 23:03:00.118722 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:00.118663 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podStartSLOduration=2.118648233 podStartE2EDuration="2.118648233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:00.116866645 +0000 UTC m=+2360.739174952" watchObservedRunningTime="2026-11-03 23:03:00.118648233 +0000 UTC m=+2360.740956540" Nov 03 23:03:00.449183 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:00.449148 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:03:00.459964 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:00.459926 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 23:03:00.460268 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:00.460236 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 23:03:00.977704 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:00.977665 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 23:03:00.978605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:00.978556 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 23:03:01.450829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:01.448195 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 23:03:01.450829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:01.448257 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 23:03:01.979248 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:01.979207 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 23:03:01.979849 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:01.979788 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 23:03:02.467827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:02.467367 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 23:03:02.467827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:02.467460 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 23:03:02.974827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:02.974132 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 23:03:02.974827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:02.974204 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 23:03:03.451097 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:03.451058 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 23:03:03.451355 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:03.451324 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 23:03:03.888825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:03.886935 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v"] Nov 03 23:03:03.986848 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:03.985976 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 23:03:03.986848 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:03.986037 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 23:03:04.491827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:04.481122 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 23:03:04.491827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:04.481211 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 23:03:05.086824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:05.086028 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 23:03:05.086824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:05.086121 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 23:03:05.454880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:05.454216 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 03 23:03:05.454880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:05.454282 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 03 23:03:05.983402 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:05.983353 5272 patch_prober.go:29] interesting pod/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Nov 03 23:03:05.983623 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:05.983437 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Nov 03 23:03:06.951565 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:06.951521 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:03:06.952633 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:06.952597 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:03:06.954012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:06.953989 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:03:07.194138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:07.194105 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv"] Nov 03 23:03:07.197790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:07.197756 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-545b5d-n6vlv"] Nov 03 23:03:07.401191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:07.401148 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:03:07.401637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:07.401608 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:03:07.402739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:07.402715 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:03:07.494527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:07.494481 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec313b2b-e4c0-460e-ac49-464b8e4f80e5" path="/var/lib/kubelet/pods/ec313b2b-e4c0-460e-ac49-464b8e4f80e5/volumes" Nov 03 23:03:08.403884 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:08.403848 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw"] Nov 03 23:03:08.408709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:08.408681 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee74cb-c7prw"] Nov 03 23:03:08.887344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:08.887295 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" podUID="426df26b-3a98-4e13-a910-9985cd979aca" containerName="payload-processing" containerID="cri-o://9ca4f5253a2fad2eff4369ddfdccc6b85e7512cddba46965d8a3954b14bb6242" gracePeriod=25 Nov 03 23:03:09.181790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.181756 5272 generic.go:356] "Generic (PLEG): container finished" podID="426df26b-3a98-4e13-a910-9985cd979aca" containerID="9ca4f5253a2fad2eff4369ddfdccc6b85e7512cddba46965d8a3954b14bb6242" exitCode=0 Nov 03 23:03:09.181978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.181834 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" event={"ID":"426df26b-3a98-4e13-a910-9985cd979aca","Type":"ContainerDied","Data":"9ca4f5253a2fad2eff4369ddfdccc6b85e7512cddba46965d8a3954b14bb6242"} Nov 03 23:03:09.206954 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.206926 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 23:03:09.279768 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.279743 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426df26b-3a98-4e13-a910-9985cd979aca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426df26b-3a98-4e13-a910-9985cd979aca-plugins-config-volume\") pod \"426df26b-3a98-4e13-a910-9985cd979aca\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " Nov 03 23:03:09.279951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.279872 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426df26b-3a98-4e13-a910-9985cd979aca-kube-api-access-mvqxd\" (UniqueName: \"kubernetes.io/projected/426df26b-3a98-4e13-a910-9985cd979aca-kube-api-access-mvqxd\") pod \"426df26b-3a98-4e13-a910-9985cd979aca\" (UID: \"426df26b-3a98-4e13-a910-9985cd979aca\") " Nov 03 23:03:09.280149 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.280124 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426df26b-3a98-4e13-a910-9985cd979aca-plugins-config-volume" pod "426df26b-3a98-4e13-a910-9985cd979aca" (UID: "426df26b-3a98-4e13-a910-9985cd979aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:09.280404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.280374 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426df26b-3a98-4e13-a910-9985cd979aca-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:09.282080 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.282059 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426df26b-3a98-4e13-a910-9985cd979aca-kube-api-access-mvqxd" pod "426df26b-3a98-4e13-a910-9985cd979aca" (UID: "426df26b-3a98-4e13-a910-9985cd979aca"). InnerVolumeSpecName "kube-api-access-mvqxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:09.381550 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.381508 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvqxd\" (UniqueName: \"kubernetes.io/projected/426df26b-3a98-4e13-a910-9985cd979aca-kube-api-access-mvqxd\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:09.499721 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:09.499633 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263c14ec-4fdf-48c0-87ac-2330164d2b92" path="/var/lib/kubelet/pods/263c14ec-4fdf-48c0-87ac-2330164d2b92/volumes" Nov 03 23:03:10.191510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:10.191428 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" Nov 03 23:03:10.191510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:10.191444 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v" event={"ID":"426df26b-3a98-4e13-a910-9985cd979aca","Type":"ContainerDied","Data":"1697beb6b903667b440ff134b5663a2efb9375ad88e82b43557e5c69aeef47bd"} Nov 03 23:03:10.191510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:10.191509 5272 scope.go:122] "RemoveContainer" containerID="9ca4f5253a2fad2eff4369ddfdccc6b85e7512cddba46965d8a3954b14bb6242" Nov 03 23:03:10.390427 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:10.390396 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v"] Nov 03 23:03:10.394927 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:10.394903 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-18c03a-85c44654fd-b8t9v"] Nov 03 23:03:11.493602 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:11.493564 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426df26b-3a98-4e13-a910-9985cd979aca" path="/var/lib/kubelet/pods/426df26b-3a98-4e13-a910-9985cd979aca/volumes" Nov 03 23:03:12.164381 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.164330 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" containerID="cri-o://432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f" gracePeriod=45 Nov 03 23:03:12.187211 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.187180 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" containerID="cri-o://1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2" gracePeriod=45 Nov 03 23:03:12.475783 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.475754 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 23:03:12.479201 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.479182 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 23:03:12.618393 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618357 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-kube-api-access-sbnkc\" (UniqueName: \"kubernetes.io/projected/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-kube-api-access-sbnkc\") pod \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " Nov 03 23:03:12.618393 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618399 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-kube-api-access-2txzw\" (UniqueName: \"kubernetes.io/projected/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-kube-api-access-2txzw\") pod \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618459 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tls-certs\" (UniqueName: \"kubernetes.io/secret/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tls-certs\") pod \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618495 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tmp-dir\") pod \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618546 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-home\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-home\") pod \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618619 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-home\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-home\") pod \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618667 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tmp-dir\") pod \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618709 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-model-cache\") pod \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618786 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tls-certs\") pod \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618828 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-dshm\") pod \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618898 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-dshm\") pod \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\" (UID: \"acae9a3d-2b1b-40e7-8a97-2a52d32a91da\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.618943 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-model-cache\") pod \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\" (UID: \"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49\") " Nov 03 23:03:12.619110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.619010 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tmp-dir" pod "acae9a3d-2b1b-40e7-8a97-2a52d32a91da" (UID: "acae9a3d-2b1b-40e7-8a97-2a52d32a91da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.620043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.619166 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-home" pod "acae9a3d-2b1b-40e7-8a97-2a52d32a91da" (UID: "acae9a3d-2b1b-40e7-8a97-2a52d32a91da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.620043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.619253 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-home" pod "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" (UID: "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.620043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.619415 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tmp-dir" pod "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" (UID: "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.620043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.619428 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-model-cache" pod "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" (UID: "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.620043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.619674 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-model-cache" pod "acae9a3d-2b1b-40e7-8a97-2a52d32a91da" (UID: "acae9a3d-2b1b-40e7-8a97-2a52d32a91da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.620316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.620081 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.620316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.620103 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.620316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.620116 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.620316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.620128 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.620316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.620141 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.620316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.620157 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.625352 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.625302 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-kube-api-access-sbnkc" pod "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" (UID: "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49"). InnerVolumeSpecName "kube-api-access-sbnkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:12.625503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.625476 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tls-certs" pod "acae9a3d-2b1b-40e7-8a97-2a52d32a91da" (UID: "acae9a3d-2b1b-40e7-8a97-2a52d32a91da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:12.628169 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.628142 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tls-certs" pod "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" (UID: "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:12.628271 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.628160 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-dshm" pod "acae9a3d-2b1b-40e7-8a97-2a52d32a91da" (UID: "acae9a3d-2b1b-40e7-8a97-2a52d32a91da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.628271 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.628148 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-kube-api-access-2txzw" pod "acae9a3d-2b1b-40e7-8a97-2a52d32a91da" (UID: "acae9a3d-2b1b-40e7-8a97-2a52d32a91da"). InnerVolumeSpecName "kube-api-access-2txzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:12.629319 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.629296 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-dshm" pod "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" (UID: "ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:12.722865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.722752 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.722865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.722826 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.722865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.722843 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.722865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.722855 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnkc\" (UniqueName: \"kubernetes.io/projected/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49-kube-api-access-sbnkc\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.723202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.722871 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txzw\" (UniqueName: \"kubernetes.io/projected/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-kube-api-access-2txzw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:12.723202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:12.722902 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acae9a3d-2b1b-40e7-8a97-2a52d32a91da-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:13.216299 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.216259 5272 generic.go:356] "Generic (PLEG): container finished" podID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerID="432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f" exitCode=0 Nov 03 23:03:13.216637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.216620 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" Nov 03 23:03:13.217321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.217291 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" event={"ID":"acae9a3d-2b1b-40e7-8a97-2a52d32a91da","Type":"ContainerDied","Data":"432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f"} Nov 03 23:03:13.217457 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.217338 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp" event={"ID":"acae9a3d-2b1b-40e7-8a97-2a52d32a91da","Type":"ContainerDied","Data":"4e485f473497142bd18b5144eb13e0bf5b91f53a742e1bd3df1d47a6ebd4a15a"} Nov 03 23:03:13.217457 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.217360 5272 scope.go:122] "RemoveContainer" containerID="432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f" Nov 03 23:03:13.219548 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.219523 5272 generic.go:356] "Generic (PLEG): container finished" podID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerID="1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2" exitCode=0 Nov 03 23:03:13.219672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.219597 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" event={"ID":"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49","Type":"ContainerDied","Data":"1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2"} Nov 03 23:03:13.219672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.219624 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" event={"ID":"ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49","Type":"ContainerDied","Data":"464c5198658eb74e266000e306a504f97350be7c35b066b4e43813334ca6f5a5"} Nov 03 23:03:13.219792 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.219681 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh" Nov 03 23:03:13.229913 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.229895 5272 scope.go:122] "RemoveContainer" containerID="432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f" Nov 03 23:03:13.230344 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:03:13.230321 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f\": container with ID starting with 432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f not found: ID does not exist" containerID="432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f" Nov 03 23:03:13.230448 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.230350 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f"} err="failed to get container status \"432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f\": rpc error: code = NotFound desc = could not find container \"432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f\": container with ID starting with 432da7f74f96015298e1e83bce8125a050ade1aa0919d05c87d2628f539f197f not found: ID does not exist" Nov 03 23:03:13.230448 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.230368 5272 scope.go:122] "RemoveContainer" containerID="1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2" Nov 03 23:03:13.241224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.241210 5272 scope.go:122] "RemoveContainer" containerID="1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2" Nov 03 23:03:13.241463 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:03:13.241446 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2\": container with ID starting with 1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2 not found: ID does not exist" containerID="1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2" Nov 03 23:03:13.241521 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.241469 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2"} err="failed to get container status \"1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2\": rpc error: code = NotFound desc = could not find container \"1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2\": container with ID starting with 1af34efd277020f9e855ede66ae1db833edcc03f6bade9ae6e5fc8d2993b32c2 not found: ID does not exist" Nov 03 23:03:13.373436 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.373399 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp"] Nov 03 23:03:13.377024 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.376992 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/distinct-2-w4-18c03a-kserve-6c4bf45848-zz7pp"] Nov 03 23:03:13.477461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.477376 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh"] Nov 03 23:03:13.494647 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.494613 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" path="/var/lib/kubelet/pods/acae9a3d-2b1b-40e7-8a97-2a52d32a91da/volumes" Nov 03 23:03:13.495017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:13.494997 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-18c03a/simulator-w4-18c03a-kserve-6f7d85fd57-zl7nh"] Nov 03 23:03:15.493519 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:15.493488 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" path="/var/lib/kubelet/pods/ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49/volumes" Nov 03 23:03:15.890378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:15.890344 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77"] Nov 03 23:03:15.890551 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:15.890534 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" containerID="cri-o://4c96b3a826da721040a6942131fd7d1dd5eb60e16bf10e38cb6939fae4a714f0" gracePeriod=30 Nov 03 23:03:19.414313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.414200 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm"] Nov 03 23:03:19.415091 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415065 5272 cpu_manager.go:414] "containerMap: removing container" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" Nov 03 23:03:19.415091 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415091 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" Nov 03 23:03:19.415257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415120 5272 cpu_manager.go:414] "containerMap: removing container" podUID="426df26b-3a98-4e13-a910-9985cd979aca" containerName="payload-processing" Nov 03 23:03:19.415257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415126 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426df26b-3a98-4e13-a910-9985cd979aca" containerName="payload-processing" Nov 03 23:03:19.415257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415137 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" Nov 03 23:03:19.415257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415143 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" Nov 03 23:03:19.415257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415227 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="acae9a3d-2b1b-40e7-8a97-2a52d32a91da" containerName="main" Nov 03 23:03:19.415257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415240 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="426df26b-3a98-4e13-a910-9985cd979aca" containerName="payload-processing" Nov 03 23:03:19.415257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.415246 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff43529e-2bec-4c7e-b8d4-a2c0b3d14a49" containerName="main" Nov 03 23:03:19.420573 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.420551 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.467508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.467476 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm"] Nov 03 23:03:19.469523 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.469498 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:03:19.510075 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.510043 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:03:19.510075 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.510054 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:03:19.515632 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.515619 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:03:19.519123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.519103 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g"] Nov 03 23:03:19.523363 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.523347 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:19.538281 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.538255 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g"] Nov 03 23:03:19.599825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.573966 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.599825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.592602 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.599825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.592696 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mhx\" (UniqueName: \"kubernetes.io/projected/1c19def4-abcb-480f-95ac-b61e5c19c650-kube-api-access-q9mhx\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.599825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.593444 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ac6633-dockercfg-sk5qx\"" Nov 03 23:03:19.599825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.594427 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ac6633\"" Nov 03 23:03:19.648919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.648886 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd"] Nov 03 23:03:19.666556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.666480 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd"] Nov 03 23:03:19.666899 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.666878 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:19.693920 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.693886 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0851de15-f9b3-483e-995c-3af3b18c5310-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:19.694064 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.694040 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.694122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.694101 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/0851de15-f9b3-483e-995c-3af3b18c5310-kube-api-access-2z7dr\") pod \"payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:19.694317 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.694297 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.694370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.694334 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mhx\" (UniqueName: \"kubernetes.io/projected/1c19def4-abcb-480f-95ac-b61e5c19c650-kube-api-access-q9mhx\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.694910 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.694883 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.696819 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.696777 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.751717 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.751691 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mhx\" (UniqueName: \"kubernetes.io/projected/1c19def4-abcb-480f-95ac-b61e5c19c650-kube-api-access-q9mhx\") pod \"maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:19.795573 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.795552 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ntz\" (UniqueName: \"kubernetes.io/projected/942b4c66-7202-4e1a-ae7f-dba0b94de655-kube-api-access-56ntz\") pod \"payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:19.795657 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.795581 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0851de15-f9b3-483e-995c-3af3b18c5310-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:19.795657 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.795618 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b4c66-7202-4e1a-ae7f-dba0b94de655-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:19.795740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.795702 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/0851de15-f9b3-483e-995c-3af3b18c5310-kube-api-access-2z7dr\") pod \"payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:19.796267 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.796249 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0851de15-f9b3-483e-995c-3af3b18c5310-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:19.852819 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.852785 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/0851de15-f9b3-483e-995c-3af3b18c5310-kube-api-access-2z7dr\") pod \"payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:19.897164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.897139 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b4c66-7202-4e1a-ae7f-dba0b94de655-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:19.897298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.897286 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ntz\" (UniqueName: \"kubernetes.io/projected/942b4c66-7202-4e1a-ae7f-dba0b94de655-kube-api-access-56ntz\") pod \"payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:19.897754 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.897737 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b4c66-7202-4e1a-ae7f-dba0b94de655-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:19.954875 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.954820 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ntz\" (UniqueName: \"kubernetes.io/projected/942b4c66-7202-4e1a-ae7f-dba0b94de655-kube-api-access-56ntz\") pod \"payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:19.989647 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:19.989623 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:20.034973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:20.034946 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:20.136978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:20.136946 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:20.398828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:20.396880 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd"] Nov 03 23:03:20.450057 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:20.450003 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm"] Nov 03 23:03:20.457861 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:03:20.453929 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942b4c66_7202_4e1a_ae7f_dba0b94de655.slice/crio-0cb170e0b15717773c451f6daa9ac83c1d614bd9aec76a013814983f490e4b81 WatchSource:0}: Error finding container 0cb170e0b15717773c451f6daa9ac83c1d614bd9aec76a013814983f490e4b81: Status 404 returned error can't find the container with id 0cb170e0b15717773c451f6daa9ac83c1d614bd9aec76a013814983f490e4b81 Nov 03 23:03:20.540086 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:03:20.539746 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0851de15_f9b3_483e_995c_3af3b18c5310.slice/crio-8b022e9c47744b6ef14f1dd1ca3e26b2e7d131b09f3e0ebe3fdab7894615ccf7 WatchSource:0}: Error finding container 8b022e9c47744b6ef14f1dd1ca3e26b2e7d131b09f3e0ebe3fdab7894615ccf7: Status 404 returned error can't find the container with id 8b022e9c47744b6ef14f1dd1ca3e26b2e7d131b09f3e0ebe3fdab7894615ccf7 Nov 03 23:03:20.550423 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:20.550242 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g"] Nov 03 23:03:20.655990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:20.655953 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:03:21.289502 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.289469 5272 generic.go:356] "Generic (PLEG): container finished" podID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerID="4c96b3a826da721040a6942131fd7d1dd5eb60e16bf10e38cb6939fae4a714f0" exitCode=0 Nov 03 23:03:21.289664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.289589 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" event={"ID":"a5f505e0-4538-4cd8-b040-7910b2f8d54e","Type":"ContainerDied","Data":"4c96b3a826da721040a6942131fd7d1dd5eb60e16bf10e38cb6939fae4a714f0"} Nov 03 23:03:21.289664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.289660 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" event={"ID":"a5f505e0-4538-4cd8-b040-7910b2f8d54e","Type":"ContainerDied","Data":"21a960b00ec8adca06fc8314e370018ff55289f1dee4502c863e376e695666b0"} Nov 03 23:03:21.289819 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.289680 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a960b00ec8adca06fc8314e370018ff55289f1dee4502c863e376e695666b0" Nov 03 23:03:21.291477 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.291445 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" event={"ID":"0851de15-f9b3-483e-995c-3af3b18c5310","Type":"ContainerStarted","Data":"2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee"} Nov 03 23:03:21.291608 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.291501 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" event={"ID":"0851de15-f9b3-483e-995c-3af3b18c5310","Type":"ContainerStarted","Data":"8b022e9c47744b6ef14f1dd1ca3e26b2e7d131b09f3e0ebe3fdab7894615ccf7"} Nov 03 23:03:21.291608 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.291534 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:21.292533 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.292508 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:03:21.293153 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.293130 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" event={"ID":"1c19def4-abcb-480f-95ac-b61e5c19c650","Type":"ContainerStarted","Data":"87bfd7583663f397e289cf6b22f8fbf0820bd83777a2175cfa13f18667676f69"} Nov 03 23:03:21.293247 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.293188 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" event={"ID":"1c19def4-abcb-480f-95ac-b61e5c19c650","Type":"ContainerStarted","Data":"c99fa9380123a7d7a9a71657c0b90b50c7e0ee54e27f6d720345cee6abcebb9c"} Nov 03 23:03:21.294673 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.294651 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" event={"ID":"942b4c66-7202-4e1a-ae7f-dba0b94de655","Type":"ContainerStarted","Data":"9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27"} Nov 03 23:03:21.294777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.294689 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" event={"ID":"942b4c66-7202-4e1a-ae7f-dba0b94de655","Type":"ContainerStarted","Data":"0cb170e0b15717773c451f6daa9ac83c1d614bd9aec76a013814983f490e4b81"} Nov 03 23:03:21.294884 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.294814 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:21.326272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326238 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-podinfo\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326379 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326282 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-credential-socket\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326379 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326307 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-socket\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326379 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326342 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istiod-ca-cert\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326379 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326363 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-certs\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326622 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326420 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-token\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-token\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326622 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326448 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-envoy\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326622 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326480 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-data\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326663 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-credential-socket" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.326790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326680 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-socket" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.326790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326752 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-data" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.326790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326760 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-kube-api-access-bh8d8\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-kube-api-access-bh8d8\") pod \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\" (UID: \"a5f505e0-4538-4cd8-b040-7910b2f8d54e\") " Nov 03 23:03:21.326790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.326780 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istiod-ca-cert" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:21.327144 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.327117 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-certs" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.327648 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.327625 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.327782 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.327765 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.327938 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.327922 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.328043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.328029 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.328139 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.328117 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.329110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.329061 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-kube-api-access-bh8d8" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "kube-api-access-bh8d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:21.329220 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.329176 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-token" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:21.329220 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.329200 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-envoy" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:21.329529 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.329506 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-podinfo" pod "a5f505e0-4538-4cd8-b040-7910b2f8d54e" (UID: "a5f505e0-4538-4cd8-b040-7910b2f8d54e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:21.408037 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.407971 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" podStartSLOduration=2.407952114 podStartE2EDuration="2.407952114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:21.405884299 +0000 UTC m=+2382.028192607" watchObservedRunningTime="2026-11-03 23:03:21.407952114 +0000 UTC m=+2382.030260422" Nov 03 23:03:21.429881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.429799 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.429881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.429879 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.429881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.429891 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh8d8\" (UniqueName: \"kubernetes.io/projected/a5f505e0-4538-4cd8-b040-7910b2f8d54e-kube-api-access-bh8d8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.430123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.429902 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f505e0-4538-4cd8-b040-7910b2f8d54e-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:21.516095 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.516018 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" podStartSLOduration=2.516004691 podStartE2EDuration="2.516004691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:21.513549098 +0000 UTC m=+2382.135857407" watchObservedRunningTime="2026-11-03 23:03:21.516004691 +0000 UTC m=+2382.138312998" Nov 03 23:03:21.735727 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:21.735671 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" podStartSLOduration=2.7356582339999997 podStartE2EDuration="2.735658234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:21.733901067 +0000 UTC m=+2382.356209375" watchObservedRunningTime="2026-11-03 23:03:21.735658234 +0000 UTC m=+2382.357966571" Nov 03 23:03:22.305879 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:22.305846 5272 generic.go:356] "Generic (PLEG): container finished" podID="1c19def4-abcb-480f-95ac-b61e5c19c650" containerID="87bfd7583663f397e289cf6b22f8fbf0820bd83777a2175cfa13f18667676f69" exitCode=0 Nov 03 23:03:22.306065 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:22.305993 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" event={"ID":"1c19def4-abcb-480f-95ac-b61e5c19c650","Type":"ContainerDied","Data":"87bfd7583663f397e289cf6b22f8fbf0820bd83777a2175cfa13f18667676f69"} Nov 03 23:03:22.306241 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:22.306226 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77" Nov 03 23:03:22.520391 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:22.520359 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77"] Nov 03 23:03:22.566675 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:22.566601 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee74cb-openshift-default-9758876f-dwx77"] Nov 03 23:03:23.483926 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.483903 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:23.494230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.494202 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" path="/var/lib/kubelet/pods/a5f505e0-4538-4cd8-b040-7910b2f8d54e/volumes" Nov 03 23:03:23.570916 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.570837 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-openshift-ca\") pod \"1c19def4-abcb-480f-95ac-b61e5c19c650\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " Nov 03 23:03:23.571272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.570942 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c19def4-abcb-480f-95ac-b61e5c19c650-kube-api-access-q9mhx\" (UniqueName: \"kubernetes.io/projected/1c19def4-abcb-480f-95ac-b61e5c19c650-kube-api-access-q9mhx\") pod \"1c19def4-abcb-480f-95ac-b61e5c19c650\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " Nov 03 23:03:23.571272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.571017 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-xks-ca\") pod \"1c19def4-abcb-480f-95ac-b61e5c19c650\" (UID: \"1c19def4-abcb-480f-95ac-b61e5c19c650\") " Nov 03 23:03:23.571272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.571224 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-openshift-ca" pod "1c19def4-abcb-480f-95ac-b61e5c19c650" (UID: "1c19def4-abcb-480f-95ac-b61e5c19c650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:23.571534 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.571514 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.573147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.573111 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-xks-ca" pod "1c19def4-abcb-480f-95ac-b61e5c19c650" (UID: "1c19def4-abcb-480f-95ac-b61e5c19c650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:23.573147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.573135 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c19def4-abcb-480f-95ac-b61e5c19c650-kube-api-access-q9mhx" pod "1c19def4-abcb-480f-95ac-b61e5c19c650" (UID: "1c19def4-abcb-480f-95ac-b61e5c19c650"). InnerVolumeSpecName "kube-api-access-q9mhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:23.673276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.673239 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mhx\" (UniqueName: \"kubernetes.io/projected/1c19def4-abcb-480f-95ac-b61e5c19c650-kube-api-access-q9mhx\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.673276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:23.673271 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c19def4-abcb-480f-95ac-b61e5c19c650-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:24.322221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:24.322189 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" Nov 03 23:03:24.322221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:24.322207 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm" event={"ID":"1c19def4-abcb-480f-95ac-b61e5c19c650","Type":"ContainerDied","Data":"c99fa9380123a7d7a9a71657c0b90b50c7e0ee54e27f6d720345cee6abcebb9c"} Nov 03 23:03:24.322433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:24.322280 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c99fa9380123a7d7a9a71657c0b90b50c7e0ee54e27f6d720345cee6abcebb9c" Nov 03 23:03:24.500734 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:03:24.500700 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:03:24.775314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:24.775232 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5"] Nov 03 23:03:24.775795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:24.775480 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" containerID="cri-o://d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2" gracePeriod=30 Nov 03 23:03:24.809556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:24.809526 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt"] Nov 03 23:03:24.809767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:24.809744 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" containerID="cri-o://1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba" gracePeriod=30 Nov 03 23:03:28.562081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.562048 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf"] Nov 03 23:03:28.562644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.562623 5272 cpu_manager.go:414] "containerMap: removing container" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" Nov 03 23:03:28.562644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.562638 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" Nov 03 23:03:28.562732 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.562650 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1c19def4-abcb-480f-95ac-b61e5c19c650" containerName="revoke-keys" Nov 03 23:03:28.562732 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.562657 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c19def4-abcb-480f-95ac-b61e5c19c650" containerName="revoke-keys" Nov 03 23:03:28.562900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.562738 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f505e0-4538-4cd8-b040-7910b2f8d54e" containerName="istio-proxy" Nov 03 23:03:28.562900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.562746 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c19def4-abcb-480f-95ac-b61e5c19c650" containerName="revoke-keys" Nov 03 23:03:28.567670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.567651 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.620414 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.620390 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf"] Nov 03 23:03:28.620739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.620718 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:03:28.652953 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.652925 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.653108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.653079 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.653344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.653325 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n44j\" (UniqueName: \"kubernetes.io/projected/18342e26-c736-4b45-b358-aa5f9e1ed575-kube-api-access-6n44j\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.663667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.663647 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:03:28.663667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.663646 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:03:28.663824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.663669 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:03:28.670433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.670413 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn"] Nov 03 23:03:28.674440 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.674425 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.693218 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.693195 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn"] Nov 03 23:03:28.755145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.755109 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.755298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.755166 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27lv\" (UniqueName: \"kubernetes.io/projected/9fbe10fe-38b5-4888-a5c6-759ba0006c63-kube-api-access-h27lv\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.755343 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.755297 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.755343 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.755323 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.755449 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.755352 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.755449 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.755404 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n44j\" (UniqueName: \"kubernetes.io/projected/18342e26-c736-4b45-b358-aa5f9e1ed575-kube-api-access-6n44j\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.756012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.755987 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.757620 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.757601 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.812898 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.812843 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n44j\" (UniqueName: \"kubernetes.io/projected/18342e26-c736-4b45-b358-aa5f9e1ed575-kube-api-access-6n44j\") pod \"maas-api-revoke-sub-728a5e13-zmkrf\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.856434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.856407 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.856580 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.856517 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.856580 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.856555 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27lv\" (UniqueName: \"kubernetes.io/projected/9fbe10fe-38b5-4888-a5c6-759ba0006c63-kube-api-access-h27lv\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.857183 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.857160 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.858644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.858615 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.881641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.881622 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:28.913766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.913740 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27lv\" (UniqueName: \"kubernetes.io/projected/9fbe10fe-38b5-4888-a5c6-759ba0006c63-kube-api-access-h27lv\") pod \"maas-api-revoke-sub-12b61084-kq4pn\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:28.988298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:28.988260 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:29.257045 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:29.256977 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf"] Nov 03 23:03:29.262542 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:03:29.262508 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18342e26_c736_4b45_b358_aa5f9e1ed575.slice/crio-57b0dd908fd1f66f571c5bf77fd5bce8e6a40f9e5bff39329f5514ee49aedf19 WatchSource:0}: Error finding container 57b0dd908fd1f66f571c5bf77fd5bce8e6a40f9e5bff39329f5514ee49aedf19: Status 404 returned error can't find the container with id 57b0dd908fd1f66f571c5bf77fd5bce8e6a40f9e5bff39329f5514ee49aedf19 Nov 03 23:03:29.360168 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:29.360140 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" event={"ID":"18342e26-c736-4b45-b358-aa5f9e1ed575","Type":"ContainerStarted","Data":"57b0dd908fd1f66f571c5bf77fd5bce8e6a40f9e5bff39329f5514ee49aedf19"} Nov 03 23:03:29.371899 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:29.371867 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn"] Nov 03 23:03:29.377478 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:03:29.377455 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbe10fe_38b5_4888_a5c6_759ba0006c63.slice/crio-ffcb306680924b28a60ff8d047949aba7286b3a6f0b488dae7d7dbaa2ed90c4e WatchSource:0}: Error finding container ffcb306680924b28a60ff8d047949aba7286b3a6f0b488dae7d7dbaa2ed90c4e: Status 404 returned error can't find the container with id ffcb306680924b28a60ff8d047949aba7286b3a6f0b488dae7d7dbaa2ed90c4e Nov 03 23:03:29.900830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:29.900772 5272 patch_prober.go:29] interesting pod/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 23:03:29.901207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:29.900855 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 23:03:30.047185 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.047117 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn"] Nov 03 23:03:30.051570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.051545 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6284123f-zp9xn"] Nov 03 23:03:30.074682 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.074660 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 23:03:30.084029 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.084010 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:03:30.174393 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174333 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-podinfo\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.174529 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174400 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-data\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.174529 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174460 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-credential-socket\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.174529 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174510 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-envoy\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.174687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174543 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-istio-token\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-istio-token\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.174687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174593 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce868080-9616-4fec-9184-9363ec40badd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce868080-9616-4fec-9184-9363ec40badd-istiod-ca-cert\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.174687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174645 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-data" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.174687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174669 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-credential-socket\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.174687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174673 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-credential-socket" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.175006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174728 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-envoy\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.175006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174758 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-socket\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.175006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174793 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istiod-ca-cert\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.175006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174870 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-certs\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.175006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174904 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-credential-socket" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.175006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174908 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-certs\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.175006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.174958 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-data\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175015 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce868080-9616-4fec-9184-9363ec40badd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce868080-9616-4fec-9184-9363ec40badd-istio-podinfo\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175051 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-socket\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175058 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-socket" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175066 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce868080-9616-4fec-9184-9363ec40badd-istiod-ca-cert" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175100 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-certs" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175121 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-kube-api-access-rb7rw\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-kube-api-access-rb7rw\") pod \"ce868080-9616-4fec-9184-9363ec40badd\" (UID: \"ce868080-9616-4fec-9184-9363ec40badd\") " Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175178 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-kube-api-access-cc95c\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-kube-api-access-cc95c\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175221 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-token\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-token\") pod \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\" (UID: \"2a587b1b-f8dd-41f3-9766-cc48b39a60a9\") " Nov 03 23:03:30.175377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175300 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-certs" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.175844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175386 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istiod-ca-cert" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:30.175844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175505 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-data" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.175844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175596 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-socket" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175891 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175912 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175928 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175942 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175957 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175972 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175985 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.175998 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.176013 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce868080-9616-4fec-9184-9363ec40badd-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.176524 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.176028 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.177066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.177029 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-podinfo" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:30.177242 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.177218 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-istio-token" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:30.177778 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.177756 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-kube-api-access-cc95c" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "kube-api-access-cc95c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:30.177903 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.177877 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-token" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:30.178012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.177989 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-envoy" pod "2a587b1b-f8dd-41f3-9766-cc48b39a60a9" (UID: "2a587b1b-f8dd-41f3-9766-cc48b39a60a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.178135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.178116 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-envoy" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:30.178338 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.178321 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce868080-9616-4fec-9184-9363ec40badd-istio-podinfo" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:30.179164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.179146 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-kube-api-access-rb7rw" pod "ce868080-9616-4fec-9184-9363ec40badd" (UID: "ce868080-9616-4fec-9184-9363ec40badd"). InnerVolumeSpecName "kube-api-access-rb7rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:30.277726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277695 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce868080-9616-4fec-9184-9363ec40badd-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.277726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277726 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.277966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277740 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.277966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277756 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce868080-9616-4fec-9184-9363ec40badd-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.277966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277770 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7rw\" (UniqueName: \"kubernetes.io/projected/ce868080-9616-4fec-9184-9363ec40badd-kube-api-access-rb7rw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.277966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277784 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc95c\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-kube-api-access-cc95c\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.277966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277813 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.277966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.277827 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a587b1b-f8dd-41f3-9766-cc48b39a60a9-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.371656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.371623 5272 generic.go:356] "Generic (PLEG): container finished" podID="ce868080-9616-4fec-9184-9363ec40badd" containerID="d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2" exitCode=0 Nov 03 23:03:30.371783 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.371671 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" Nov 03 23:03:30.371783 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.371702 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" event={"ID":"ce868080-9616-4fec-9184-9363ec40badd","Type":"ContainerDied","Data":"d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2"} Nov 03 23:03:30.371783 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.371758 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5" event={"ID":"ce868080-9616-4fec-9184-9363ec40badd","Type":"ContainerDied","Data":"771b249e47984e753c5e67616f59e36735bec15b1bf8ed99a00935d65686b1d3"} Nov 03 23:03:30.371979 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.371787 5272 scope.go:122] "RemoveContainer" containerID="d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2" Nov 03 23:03:30.373598 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.373570 5272 generic.go:356] "Generic (PLEG): container finished" podID="18342e26-c736-4b45-b358-aa5f9e1ed575" containerID="3be5b19594c1d0b370ef6fab680acee81b3f79cc3b4e6af343b974daeb7acb13" exitCode=0 Nov 03 23:03:30.373715 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.373648 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" event={"ID":"18342e26-c736-4b45-b358-aa5f9e1ed575","Type":"ContainerDied","Data":"3be5b19594c1d0b370ef6fab680acee81b3f79cc3b4e6af343b974daeb7acb13"} Nov 03 23:03:30.375676 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.375649 5272 generic.go:356] "Generic (PLEG): container finished" podID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerID="1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba" exitCode=0 Nov 03 23:03:30.375792 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.375694 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" Nov 03 23:03:30.375792 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.375731 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" event={"ID":"2a587b1b-f8dd-41f3-9766-cc48b39a60a9","Type":"ContainerDied","Data":"1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba"} Nov 03 23:03:30.375792 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.375787 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt" event={"ID":"2a587b1b-f8dd-41f3-9766-cc48b39a60a9","Type":"ContainerDied","Data":"caf7b245cf6c127a6dcce8e105fad7fb2b0cb8f675a3746dea52951c90e9062b"} Nov 03 23:03:30.377517 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.377497 5272 generic.go:356] "Generic (PLEG): container finished" podID="9fbe10fe-38b5-4888-a5c6-759ba0006c63" containerID="9ef569df045b4cbb080058d7aad172a2491d2ceb828ab875b2d36791bc784fe1" exitCode=0 Nov 03 23:03:30.377622 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.377527 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" event={"ID":"9fbe10fe-38b5-4888-a5c6-759ba0006c63","Type":"ContainerDied","Data":"9ef569df045b4cbb080058d7aad172a2491d2ceb828ab875b2d36791bc784fe1"} Nov 03 23:03:30.377622 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.377561 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" event={"ID":"9fbe10fe-38b5-4888-a5c6-759ba0006c63","Type":"ContainerStarted","Data":"ffcb306680924b28a60ff8d047949aba7286b3a6f0b488dae7d7dbaa2ed90c4e"} Nov 03 23:03:30.395992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.395976 5272 scope.go:122] "RemoveContainer" containerID="d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2" Nov 03 23:03:30.396243 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:03:30.396222 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2\": container with ID starting with d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2 not found: ID does not exist" containerID="d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2" Nov 03 23:03:30.396284 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.396249 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2"} err="failed to get container status \"d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2\": rpc error: code = NotFound desc = could not find container \"d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2\": container with ID starting with d935ff935bb6eba34b6079a984f05a2270cfb9a3898cd6c99c09a996d36037a2 not found: ID does not exist" Nov 03 23:03:30.396284 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.396263 5272 scope.go:122] "RemoveContainer" containerID="1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba" Nov 03 23:03:30.406412 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.406393 5272 scope.go:122] "RemoveContainer" containerID="3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4" Nov 03 23:03:30.415044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.415025 5272 scope.go:122] "RemoveContainer" containerID="1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba" Nov 03 23:03:30.415302 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:03:30.415282 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba\": container with ID starting with 1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba not found: ID does not exist" containerID="1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba" Nov 03 23:03:30.415380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.415313 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba"} err="failed to get container status \"1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba\": rpc error: code = NotFound desc = could not find container \"1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba\": container with ID starting with 1f906a1b13e9e2db00b5e00853cd34ecc9e823185f1e4938134c4130c72513ba not found: ID does not exist" Nov 03 23:03:30.415380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.415335 5272 scope.go:122] "RemoveContainer" containerID="3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4" Nov 03 23:03:30.415585 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:03:30.415566 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4\": container with ID starting with 3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4 not found: ID does not exist" containerID="3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4" Nov 03 23:03:30.415626 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.415591 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4"} err="failed to get container status \"3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4\": rpc error: code = NotFound desc = could not find container \"3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4\": container with ID starting with 3b397637dc58a12c9d7aa3a5816afaa66e41266e75e853824a7393e3edad1ab4 not found: ID does not exist" Nov 03 23:03:30.527933 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.527831 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5"] Nov 03 23:03:30.540888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.540860 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-zc5t5"] Nov 03 23:03:30.642398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.642368 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt"] Nov 03 23:03:30.646406 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:30.646379 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-545b5d-openshift-default-5c8c66f65c-5ctxt"] Nov 03 23:03:31.511903 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.510986 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" path="/var/lib/kubelet/pods/2a587b1b-f8dd-41f3-9766-cc48b39a60a9/volumes" Nov 03 23:03:31.511903 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.511459 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b9b2ec-7256-424e-82e3-29e6288800d6" path="/var/lib/kubelet/pods/c6b9b2ec-7256-424e-82e3-29e6288800d6/volumes" Nov 03 23:03:31.511903 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.511878 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce868080-9616-4fec-9184-9363ec40badd" path="/var/lib/kubelet/pods/ce868080-9616-4fec-9184-9363ec40badd/volumes" Nov 03 23:03:31.571087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.571064 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:31.610788 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.610764 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:31.698166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.698091 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbe10fe-38b5-4888-a5c6-759ba0006c63-kube-api-access-h27lv\" (UniqueName: \"kubernetes.io/projected/9fbe10fe-38b5-4888-a5c6-759ba0006c63-kube-api-access-h27lv\") pod \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " Nov 03 23:03:31.698333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.698241 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-xks-ca\") pod \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " Nov 03 23:03:31.698333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.698283 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-openshift-ca\") pod \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\" (UID: \"9fbe10fe-38b5-4888-a5c6-759ba0006c63\") " Nov 03 23:03:31.698631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.698607 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-openshift-ca" pod "9fbe10fe-38b5-4888-a5c6-759ba0006c63" (UID: "9fbe10fe-38b5-4888-a5c6-759ba0006c63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:31.698972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.698952 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:31.700254 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.700228 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbe10fe-38b5-4888-a5c6-759ba0006c63-kube-api-access-h27lv" pod "9fbe10fe-38b5-4888-a5c6-759ba0006c63" (UID: "9fbe10fe-38b5-4888-a5c6-759ba0006c63"). InnerVolumeSpecName "kube-api-access-h27lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:31.700378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.700293 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-xks-ca" pod "9fbe10fe-38b5-4888-a5c6-759ba0006c63" (UID: "9fbe10fe-38b5-4888-a5c6-759ba0006c63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:31.800160 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.800127 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-xks-ca\") pod \"18342e26-c736-4b45-b358-aa5f9e1ed575\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " Nov 03 23:03:31.800366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.800337 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-openshift-ca\") pod \"18342e26-c736-4b45-b358-aa5f9e1ed575\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " Nov 03 23:03:31.800467 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.800377 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18342e26-c736-4b45-b358-aa5f9e1ed575-kube-api-access-6n44j\" (UniqueName: \"kubernetes.io/projected/18342e26-c736-4b45-b358-aa5f9e1ed575-kube-api-access-6n44j\") pod \"18342e26-c736-4b45-b358-aa5f9e1ed575\" (UID: \"18342e26-c736-4b45-b358-aa5f9e1ed575\") " Nov 03 23:03:31.800760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.800731 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-openshift-ca" pod "18342e26-c736-4b45-b358-aa5f9e1ed575" (UID: "18342e26-c736-4b45-b358-aa5f9e1ed575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:31.801007 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.800983 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h27lv\" (UniqueName: \"kubernetes.io/projected/9fbe10fe-38b5-4888-a5c6-759ba0006c63-kube-api-access-h27lv\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:31.801078 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.801015 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbe10fe-38b5-4888-a5c6-759ba0006c63-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:31.801078 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.801032 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:31.802563 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.802535 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18342e26-c736-4b45-b358-aa5f9e1ed575-kube-api-access-6n44j" pod "18342e26-c736-4b45-b358-aa5f9e1ed575" (UID: "18342e26-c736-4b45-b358-aa5f9e1ed575"). InnerVolumeSpecName "kube-api-access-6n44j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:31.802660 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.802579 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-xks-ca" pod "18342e26-c736-4b45-b358-aa5f9e1ed575" (UID: "18342e26-c736-4b45-b358-aa5f9e1ed575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:31.901963 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.901928 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18342e26-c736-4b45-b358-aa5f9e1ed575-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:31.901963 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:31.901959 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n44j\" (UniqueName: \"kubernetes.io/projected/18342e26-c736-4b45-b358-aa5f9e1ed575-kube-api-access-6n44j\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:32.309150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.309122 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:03:32.309456 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.309436 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:03:32.426783 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.426746 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" event={"ID":"9fbe10fe-38b5-4888-a5c6-759ba0006c63","Type":"ContainerDied","Data":"ffcb306680924b28a60ff8d047949aba7286b3a6f0b488dae7d7dbaa2ed90c4e"} Nov 03 23:03:32.427019 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.426837 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffcb306680924b28a60ff8d047949aba7286b3a6f0b488dae7d7dbaa2ed90c4e" Nov 03 23:03:32.427078 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.427020 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn" Nov 03 23:03:32.429040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.429020 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" Nov 03 23:03:32.439040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.429016 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf" event={"ID":"18342e26-c736-4b45-b358-aa5f9e1ed575","Type":"ContainerDied","Data":"57b0dd908fd1f66f571c5bf77fd5bce8e6a40f9e5bff39329f5514ee49aedf19"} Nov 03 23:03:32.439270 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:32.439249 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b0dd908fd1f66f571c5bf77fd5bce8e6a40f9e5bff39329f5514ee49aedf19" Nov 03 23:03:43.463158 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:43.463128 5272 scope.go:122] "RemoveContainer" containerID="c35c0142876b35a0c5426c979892b44e66110233106dee9589df8c22ce58d2af" Nov 03 23:03:43.473453 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:43.473428 5272 scope.go:122] "RemoveContainer" containerID="d7aa15997e06f3d47a24717d4d7de21f5286210aff2160dac4cb8aba8761a227" Nov 03 23:03:43.482537 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:43.482516 5272 scope.go:122] "RemoveContainer" containerID="c1f119503b699a326cf4945dade3d693706a82c3c5c0a578d2c8087328b86803" Nov 03 23:03:43.492265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:43.492246 5272 scope.go:122] "RemoveContainer" containerID="4ac19b0ca79cb8293415acd4c96b8581189f586e6ae075486f65bb9d24e1f57d" Nov 03 23:03:54.501102 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:03:54.501067 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:03:59.218197 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:03:59.218162 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:04:19.146024 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:19.145984 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s"] Nov 03 23:04:19.192396 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:19.192365 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-18c03a-lth4s"] Nov 03 23:04:19.493362 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:19.493278 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975e5eed-7cd0-4ce1-8538-539ac45e8433" path="/var/lib/kubelet/pods/975e5eed-7cd0-4ce1-8538-539ac45e8433/volumes" Nov 03 23:04:24.502220 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:04:24.502137 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:04:28.247951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:28.247905 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:04:35.022428 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:35.022386 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm"] Nov 03 23:04:35.069202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:35.069163 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31ad7b-j2dlm"] Nov 03 23:04:35.492518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:35.492485 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c19def4-abcb-480f-95ac-b61e5c19c650" path="/var/lib/kubelet/pods/1c19def4-abcb-480f-95ac-b61e5c19c650/volumes" Nov 03 23:04:36.375957 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:36.375920 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2"] Nov 03 23:04:36.376367 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:36.376147 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" containerID="cri-o://e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51" gracePeriod=30 Nov 03 23:04:37.010224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:37.010190 5272 patch_prober.go:29] interesting pod/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:04:37.010404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:37.010245 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:04:39.934321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:39.934278 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" podUID="d79e0f53b5b82fd1a135f4636735856a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 03 23:04:41.637511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.637484 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 23:04:41.684451 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684385 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-socket\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.684451 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684425 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-credential-socket\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.684643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684456 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-kube-api-access-zt99n\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-kube-api-access-zt99n\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.684643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684560 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-envoy\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.684643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684614 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-podinfo\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.684855 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684665 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-token\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-token\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.684855 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684676 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-credential-socket" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:41.684855 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684728 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-socket" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:41.684855 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684769 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istiod-ca-cert\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.684855 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684833 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-certs\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.685147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.684930 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-data\") pod \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\" (UID: \"7fa6e3bc-8f1e-4d02-923d-16e487dd860f\") " Nov 03 23:04:41.685147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685095 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-certs" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:41.685266 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685142 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istiod-ca-cert" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:41.685349 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685322 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-data" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:41.685590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685570 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.685671 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685595 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.685671 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685612 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.685671 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685627 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.685671 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.685642 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.686933 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.686905 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-token" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:41.687049 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.686936 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-podinfo" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:04:41.687049 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.686956 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-kube-api-access-zt99n" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "kube-api-access-zt99n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:41.687049 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.686958 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-envoy" pod "7fa6e3bc-8f1e-4d02-923d-16e487dd860f" (UID: "7fa6e3bc-8f1e-4d02-923d-16e487dd860f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:41.786793 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.786757 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt99n\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-kube-api-access-zt99n\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.786793 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.786789 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.787012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.786823 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.787012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.786835 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa6e3bc-8f1e-4d02-923d-16e487dd860f-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:41.965389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.965301 5272 generic.go:356] "Generic (PLEG): container finished" podID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerID="e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51" exitCode=0 Nov 03 23:04:41.965389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.965352 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" Nov 03 23:04:41.965579 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.965383 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" event={"ID":"7fa6e3bc-8f1e-4d02-923d-16e487dd860f","Type":"ContainerDied","Data":"e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51"} Nov 03 23:04:41.965579 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.965461 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2" event={"ID":"7fa6e3bc-8f1e-4d02-923d-16e487dd860f","Type":"ContainerDied","Data":"dd4c516450d0473d0752e0ac0ebb7529a15eb76ef955db35cafdef946df13670"} Nov 03 23:04:41.965579 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.965486 5272 scope.go:122] "RemoveContainer" containerID="e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51" Nov 03 23:04:41.976179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.976159 5272 scope.go:122] "RemoveContainer" containerID="e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51" Nov 03 23:04:41.976453 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:04:41.976434 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51\": container with ID starting with e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51 not found: ID does not exist" containerID="e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51" Nov 03 23:04:41.976527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:41.976459 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51"} err="failed to get container status \"e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51\": rpc error: code = NotFound desc = could not find container \"e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51\": container with ID starting with e91d6ab4bc2a26b63a8d19f4507d12f8b2e39030ccd70ef943aae40b7dce8b51 not found: ID does not exist" Nov 03 23:04:42.124149 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:42.124116 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2"] Nov 03 23:04:42.127067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:42.127041 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18c03a-openshift-default-589bbc88df-5qmh2"] Nov 03 23:04:42.511065 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:42.511026 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2"] Nov 03 23:04:42.511273 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:42.511218 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" containerID="cri-o://2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce" gracePeriod=30 Nov 03 23:04:43.422468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.422432 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj"] Nov 03 23:04:43.423095 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423076 5272 cpu_manager.go:414] "containerMap: removing container" podUID="18342e26-c736-4b45-b358-aa5f9e1ed575" containerName="revoke-keys" Nov 03 23:04:43.423095 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423094 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18342e26-c736-4b45-b358-aa5f9e1ed575" containerName="revoke-keys" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423115 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423123 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423148 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423153 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423170 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423175 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423197 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9fbe10fe-38b5-4888-a5c6-759ba0006c63" containerName="revoke-keys" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423205 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbe10fe-38b5-4888-a5c6-759ba0006c63" containerName="revoke-keys" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423219 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" Nov 03 23:04:43.423224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423225 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" Nov 03 23:04:43.423852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423321 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce868080-9616-4fec-9184-9363ec40badd" containerName="istio-proxy" Nov 03 23:04:43.423852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423338 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbe10fe-38b5-4888-a5c6-759ba0006c63" containerName="revoke-keys" Nov 03 23:04:43.423852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423347 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" Nov 03 23:04:43.423852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423356 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="18342e26-c736-4b45-b358-aa5f9e1ed575" containerName="revoke-keys" Nov 03 23:04:43.423852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.423371 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" containerName="istio-proxy" Nov 03 23:04:43.428471 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.428446 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.479463 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.479433 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:04:43.479724 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.479505 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:04:43.479864 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.479549 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:43.484410 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.484377 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj"] Nov 03 23:04:43.496094 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.496058 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa6e3bc-8f1e-4d02-923d-16e487dd860f" path="/var/lib/kubelet/pods/7fa6e3bc-8f1e-4d02-923d-16e487dd860f/volumes" Nov 03 23:04:43.505728 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.505692 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.505932 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.505742 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbkr\" (UniqueName: \"kubernetes.io/projected/71772ded-d58f-4de6-b306-64bc73c833f7-kube-api-access-7zbkr\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.506025 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.506000 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.525921 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.525896 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:04:43.556155 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.556131 5272 scope.go:122] "RemoveContainer" containerID="9a8f307ecc08e74f8a1be20afe58daf1ba7f44046c9889c7ab4658010309e9e6" Nov 03 23:04:43.567326 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.567293 5272 scope.go:122] "RemoveContainer" containerID="f2019c03e28d35a08f51dfb2bf5f31312cf15e06eea2ce2631dbb4cdb49b12ab" Nov 03 23:04:43.576415 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.576391 5272 scope.go:122] "RemoveContainer" containerID="8241c7fa9675122986f38dfc8d7d17a035858c0a3c8e7e2a5202475e87822206" Nov 03 23:04:43.606909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.606876 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.607078 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.607063 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.607137 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.607095 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbkr\" (UniqueName: \"kubernetes.io/projected/71772ded-d58f-4de6-b306-64bc73c833f7-kube-api-access-7zbkr\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.607578 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.607556 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.609518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.609494 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.665170 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.665136 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbkr\" (UniqueName: \"kubernetes.io/projected/71772ded-d58f-4de6-b306-64bc73c833f7-kube-api-access-7zbkr\") pod \"maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:43.744248 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:43.744154 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:44.106410 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:44.106372 5272 patch_prober.go:29] interesting pod/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:04:44.106583 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:44.106429 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:04:44.147533 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:44.147493 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj"] Nov 03 23:04:44.153796 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:04:44.153767 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71772ded_d58f_4de6_b306_64bc73c833f7.slice/crio-2bc2e7ef920e63712ae7c769fe564bb69bddbace8112e3a2b6c01bb43a7996b7 WatchSource:0}: Error finding container 2bc2e7ef920e63712ae7c769fe564bb69bddbace8112e3a2b6c01bb43a7996b7: Status 404 returned error can't find the container with id 2bc2e7ef920e63712ae7c769fe564bb69bddbace8112e3a2b6c01bb43a7996b7 Nov 03 23:04:44.989833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:44.989759 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" event={"ID":"71772ded-d58f-4de6-b306-64bc73c833f7","Type":"ContainerStarted","Data":"cf6f34144c9dd9e9b779c0321dca552334a823c17d8156688eff3db0a8ec133a"} Nov 03 23:04:44.990263 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:44.989857 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" event={"ID":"71772ded-d58f-4de6-b306-64bc73c833f7","Type":"ContainerStarted","Data":"2bc2e7ef920e63712ae7c769fe564bb69bddbace8112e3a2b6c01bb43a7996b7"} Nov 03 23:04:45.099722 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:45.099669 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" podStartSLOduration=2.099653441 podStartE2EDuration="2.099653441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:45.098142928 +0000 UTC m=+2465.720451241" watchObservedRunningTime="2026-11-03 23:04:45.099653441 +0000 UTC m=+2465.721961749" Nov 03 23:04:45.998072 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:45.998036 5272 generic.go:356] "Generic (PLEG): container finished" podID="71772ded-d58f-4de6-b306-64bc73c833f7" containerID="cf6f34144c9dd9e9b779c0321dca552334a823c17d8156688eff3db0a8ec133a" exitCode=0 Nov 03 23:04:45.998496 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:45.998126 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" event={"ID":"71772ded-d58f-4de6-b306-64bc73c833f7","Type":"ContainerDied","Data":"cf6f34144c9dd9e9b779c0321dca552334a823c17d8156688eff3db0a8ec133a"} Nov 03 23:04:47.140683 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.140656 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:47.245996 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.245950 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71772ded-d58f-4de6-b306-64bc73c833f7-kube-api-access-7zbkr\" (UniqueName: \"kubernetes.io/projected/71772ded-d58f-4de6-b306-64bc73c833f7-kube-api-access-7zbkr\") pod \"71772ded-d58f-4de6-b306-64bc73c833f7\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " Nov 03 23:04:47.246188 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.246141 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-openshift-ca\") pod \"71772ded-d58f-4de6-b306-64bc73c833f7\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " Nov 03 23:04:47.246188 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.246178 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-xks-ca\") pod \"71772ded-d58f-4de6-b306-64bc73c833f7\" (UID: \"71772ded-d58f-4de6-b306-64bc73c833f7\") " Nov 03 23:04:47.246550 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.246520 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-openshift-ca" pod "71772ded-d58f-4de6-b306-64bc73c833f7" (UID: "71772ded-d58f-4de6-b306-64bc73c833f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:47.248075 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.248041 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71772ded-d58f-4de6-b306-64bc73c833f7-kube-api-access-7zbkr" pod "71772ded-d58f-4de6-b306-64bc73c833f7" (UID: "71772ded-d58f-4de6-b306-64bc73c833f7"). InnerVolumeSpecName "kube-api-access-7zbkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:47.248166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.248140 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-xks-ca" pod "71772ded-d58f-4de6-b306-64bc73c833f7" (UID: "71772ded-d58f-4de6-b306-64bc73c833f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:47.347412 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.347377 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.347412 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.347407 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71772ded-d58f-4de6-b306-64bc73c833f7-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.347605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.347421 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbkr\" (UniqueName: \"kubernetes.io/projected/71772ded-d58f-4de6-b306-64bc73c833f7-kube-api-access-7zbkr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.745364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.745336 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 23:04:47.853306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853275 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-kube-api-access-7lslb\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-kube-api-access-7lslb\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853314 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-envoy\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853359 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9060091-788b-47ed-8934-18fc8c9bc462-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9060091-788b-47ed-8934-18fc8c9bc462-istio-podinfo\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853384 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-istio-token\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-istio-token\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853457 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9060091-788b-47ed-8934-18fc8c9bc462-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9060091-788b-47ed-8934-18fc8c9bc462-istiod-ca-cert\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853493 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-certs\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853541 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-data\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853604 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-socket\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.853917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.853721 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-credential-socket\") pod \"f9060091-788b-47ed-8934-18fc8c9bc462\" (UID: \"f9060091-788b-47ed-8934-18fc8c9bc462\") " Nov 03 23:04:47.854140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.854100 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9060091-788b-47ed-8934-18fc8c9bc462-istiod-ca-cert" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:47.854213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.854139 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-socket" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:47.854267 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.854245 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9060091-788b-47ed-8934-18fc8c9bc462-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.854331 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.854266 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.854331 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.854287 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-credential-socket" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:47.854331 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.854302 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-certs" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:47.854508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.854362 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-data" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:47.855773 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.855747 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9060091-788b-47ed-8934-18fc8c9bc462-istio-podinfo" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:04:47.856087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.856061 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-envoy" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:47.856152 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.856097 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-kube-api-access-7lslb" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "kube-api-access-7lslb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:47.856195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.856149 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-istio-token" pod "f9060091-788b-47ed-8934-18fc8c9bc462" (UID: "f9060091-788b-47ed-8934-18fc8c9bc462"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:47.955110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.955078 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.955110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.955105 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lslb\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-kube-api-access-7lslb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.955290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.955118 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.955290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.955127 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9060091-788b-47ed-8934-18fc8c9bc462-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.955290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.955136 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9060091-788b-47ed-8934-18fc8c9bc462-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.955290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.955145 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:47.955290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:47.955153 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9060091-788b-47ed-8934-18fc8c9bc462-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:04:48.015354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.015315 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" event={"ID":"71772ded-d58f-4de6-b306-64bc73c833f7","Type":"ContainerDied","Data":"2bc2e7ef920e63712ae7c769fe564bb69bddbace8112e3a2b6c01bb43a7996b7"} Nov 03 23:04:48.015530 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.015382 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc2e7ef920e63712ae7c769fe564bb69bddbace8112e3a2b6c01bb43a7996b7" Nov 03 23:04:48.015530 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.015326 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj" Nov 03 23:04:48.017229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.017203 5272 generic.go:356] "Generic (PLEG): container finished" podID="f9060091-788b-47ed-8934-18fc8c9bc462" containerID="2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce" exitCode=0 Nov 03 23:04:48.017342 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.017265 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" Nov 03 23:04:48.017342 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.017286 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" event={"ID":"f9060091-788b-47ed-8934-18fc8c9bc462","Type":"ContainerDied","Data":"2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce"} Nov 03 23:04:48.017443 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.017344 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2" event={"ID":"f9060091-788b-47ed-8934-18fc8c9bc462","Type":"ContainerDied","Data":"36ca01dd93ca0c9438ca48d34f9248144b225bafeb06f093da1d819a4c68b635"} Nov 03 23:04:48.017443 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.017375 5272 scope.go:122] "RemoveContainer" containerID="2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce" Nov 03 23:04:48.028727 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.028704 5272 scope.go:122] "RemoveContainer" containerID="cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c" Nov 03 23:04:48.037881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.037862 5272 scope.go:122] "RemoveContainer" containerID="2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce" Nov 03 23:04:48.038164 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:04:48.038140 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce\": container with ID starting with 2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce not found: ID does not exist" containerID="2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce" Nov 03 23:04:48.038255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.038171 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce"} err="failed to get container status \"2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce\": rpc error: code = NotFound desc = could not find container \"2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce\": container with ID starting with 2782464e0ca1840174fdeb31a72bdc391b9c77fe5e71efd253c7a0a973dea5ce not found: ID does not exist" Nov 03 23:04:48.038255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.038188 5272 scope.go:122] "RemoveContainer" containerID="cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c" Nov 03 23:04:48.038476 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:04:48.038457 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c\": container with ID starting with cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c not found: ID does not exist" containerID="cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c" Nov 03 23:04:48.038540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.038483 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c"} err="failed to get container status \"cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c\": rpc error: code = NotFound desc = could not find container \"cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c\": container with ID starting with cd4933a88a490d555bf6b663f39e691ed6b75b7699472169d7ac3c3834dd0e9c not found: ID does not exist" Nov 03 23:04:48.172523 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.172491 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2"] Nov 03 23:04:48.178552 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:48.178526 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-31ad7b-openshift-default-65dcf6d7f7-d6fz2"] Nov 03 23:04:49.492474 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:49.492441 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" path="/var/lib/kubelet/pods/f9060091-788b-47ed-8934-18fc8c9bc462/volumes" Nov 03 23:04:54.500943 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:04:54.500908 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:04:55.561027 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:55.560992 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz"] Nov 03 23:04:55.669849 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:04:55.669781 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4"] Nov 03 23:05:00.561538 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.561478 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" podUID="95901769-38a7-4687-ae42-79dc16ab6c6a" containerName="payload-processing" containerID="cri-o://4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7" gracePeriod=25 Nov 03 23:05:00.670702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.670617 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" podUID="14fbef3b-965d-4a61-bebc-63f605f0b47f" containerName="payload-pre-processing" containerID="cri-o://29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1" gracePeriod=25 Nov 03 23:05:00.824952 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.824882 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:05:00.911651 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.911617 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95901769-38a7-4687-ae42-79dc16ab6c6a-kube-api-access-8qtzg\" (UniqueName: \"kubernetes.io/projected/95901769-38a7-4687-ae42-79dc16ab6c6a-kube-api-access-8qtzg\") pod \"95901769-38a7-4687-ae42-79dc16ab6c6a\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " Nov 03 23:05:00.911866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.911777 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95901769-38a7-4687-ae42-79dc16ab6c6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95901769-38a7-4687-ae42-79dc16ab6c6a-plugins-config-volume\") pod \"95901769-38a7-4687-ae42-79dc16ab6c6a\" (UID: \"95901769-38a7-4687-ae42-79dc16ab6c6a\") " Nov 03 23:05:00.912138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.912114 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95901769-38a7-4687-ae42-79dc16ab6c6a-plugins-config-volume" pod "95901769-38a7-4687-ae42-79dc16ab6c6a" (UID: "95901769-38a7-4687-ae42-79dc16ab6c6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:00.913918 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.913886 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95901769-38a7-4687-ae42-79dc16ab6c6a-kube-api-access-8qtzg" pod "95901769-38a7-4687-ae42-79dc16ab6c6a" (UID: "95901769-38a7-4687-ae42-79dc16ab6c6a"). InnerVolumeSpecName "kube-api-access-8qtzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:00.934261 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:00.934232 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:05:01.012642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.012596 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14fbef3b-965d-4a61-bebc-63f605f0b47f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fbef3b-965d-4a61-bebc-63f605f0b47f-plugins-config-volume\") pod \"14fbef3b-965d-4a61-bebc-63f605f0b47f\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " Nov 03 23:05:01.012895 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.012750 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fbef3b-965d-4a61-bebc-63f605f0b47f-kube-api-access-4kjf7\" (UniqueName: \"kubernetes.io/projected/14fbef3b-965d-4a61-bebc-63f605f0b47f-kube-api-access-4kjf7\") pod \"14fbef3b-965d-4a61-bebc-63f605f0b47f\" (UID: \"14fbef3b-965d-4a61-bebc-63f605f0b47f\") " Nov 03 23:05:01.013045 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.013019 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fbef3b-965d-4a61-bebc-63f605f0b47f-plugins-config-volume" pod "14fbef3b-965d-4a61-bebc-63f605f0b47f" (UID: "14fbef3b-965d-4a61-bebc-63f605f0b47f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:01.013180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.013159 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95901769-38a7-4687-ae42-79dc16ab6c6a-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:01.013180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.013178 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtzg\" (UniqueName: \"kubernetes.io/projected/95901769-38a7-4687-ae42-79dc16ab6c6a-kube-api-access-8qtzg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:01.013346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.013187 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fbef3b-965d-4a61-bebc-63f605f0b47f-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:01.014795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.014773 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fbef3b-965d-4a61-bebc-63f605f0b47f-kube-api-access-4kjf7" pod "14fbef3b-965d-4a61-bebc-63f605f0b47f" (UID: "14fbef3b-965d-4a61-bebc-63f605f0b47f"). InnerVolumeSpecName "kube-api-access-4kjf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:01.114486 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.114387 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjf7\" (UniqueName: \"kubernetes.io/projected/14fbef3b-965d-4a61-bebc-63f605f0b47f-kube-api-access-4kjf7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:01.127878 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.127845 5272 generic.go:356] "Generic (PLEG): container finished" podID="14fbef3b-965d-4a61-bebc-63f605f0b47f" containerID="29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1" exitCode=0 Nov 03 23:05:01.128052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.127883 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" Nov 03 23:05:01.128052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.127924 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" event={"ID":"14fbef3b-965d-4a61-bebc-63f605f0b47f","Type":"ContainerDied","Data":"29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1"} Nov 03 23:05:01.128052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.127981 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4" event={"ID":"14fbef3b-965d-4a61-bebc-63f605f0b47f","Type":"ContainerDied","Data":"be6a3761eb99bd231d7ca56dba8bce0c7e5af45a5c811a33a26c3934c749b682"} Nov 03 23:05:01.128052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.128009 5272 scope.go:122] "RemoveContainer" containerID="29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1" Nov 03 23:05:01.129576 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.129549 5272 generic.go:356] "Generic (PLEG): container finished" podID="95901769-38a7-4687-ae42-79dc16ab6c6a" containerID="4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7" exitCode=0 Nov 03 23:05:01.129690 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.129577 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" event={"ID":"95901769-38a7-4687-ae42-79dc16ab6c6a","Type":"ContainerDied","Data":"4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7"} Nov 03 23:05:01.129690 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.129591 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" Nov 03 23:05:01.129690 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.129630 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz" event={"ID":"95901769-38a7-4687-ae42-79dc16ab6c6a","Type":"ContainerDied","Data":"f88e42c26e9037a08120fd402c5099c3003ad573e67f257ba23418b72f3772c9"} Nov 03 23:05:01.140034 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.140008 5272 scope.go:122] "RemoveContainer" containerID="29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1" Nov 03 23:05:01.140388 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:05:01.140364 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1\": container with ID starting with 29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1 not found: ID does not exist" containerID="29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1" Nov 03 23:05:01.140450 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.140397 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1"} err="failed to get container status \"29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1\": rpc error: code = NotFound desc = could not find container \"29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1\": container with ID starting with 29f5e2f321dfd8b565946139703511fbb4f5d35c1aa6afaa4abfe945674825b1 not found: ID does not exist" Nov 03 23:05:01.140450 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.140416 5272 scope.go:122] "RemoveContainer" containerID="4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7" Nov 03 23:05:01.150247 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.150223 5272 scope.go:122] "RemoveContainer" containerID="4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7" Nov 03 23:05:01.150590 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:05:01.150565 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7\": container with ID starting with 4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7 not found: ID does not exist" containerID="4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7" Nov 03 23:05:01.150660 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.150597 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7"} err="failed to get container status \"4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7\": rpc error: code = NotFound desc = could not find container \"4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7\": container with ID starting with 4cdd4007578b5416e17e57124e25f2adb8ce901ba44f360fc740da94bb9d45f7 not found: ID does not exist" Nov 03 23:05:01.283192 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.283156 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz"] Nov 03 23:05:01.287591 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.287548 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-656ee6-7d7cf8d498-9blpz"] Nov 03 23:05:01.389575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.389487 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4"] Nov 03 23:05:01.392410 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.392379 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-656ee6-6f576464c-vsgw4"] Nov 03 23:05:01.492558 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.492523 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fbef3b-965d-4a61-bebc-63f605f0b47f" path="/var/lib/kubelet/pods/14fbef3b-965d-4a61-bebc-63f605f0b47f/volumes" Nov 03 23:05:01.492978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:01.492963 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95901769-38a7-4687-ae42-79dc16ab6c6a" path="/var/lib/kubelet/pods/95901769-38a7-4687-ae42-79dc16ab6c6a/volumes" Nov 03 23:05:09.301772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:09.301728 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:05:24.530594 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:05:24.530546 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:05:40.538988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.538951 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq"] Nov 03 23:05:40.539485 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539469 5272 cpu_manager.go:414] "containerMap: removing container" podUID="71772ded-d58f-4de6-b306-64bc73c833f7" containerName="revoke-keys" Nov 03 23:05:40.539532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539488 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71772ded-d58f-4de6-b306-64bc73c833f7" containerName="revoke-keys" Nov 03 23:05:40.539532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539519 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" Nov 03 23:05:40.539532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539527 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" Nov 03 23:05:40.539638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539547 5272 cpu_manager.go:414] "containerMap: removing container" podUID="14fbef3b-965d-4a61-bebc-63f605f0b47f" containerName="payload-pre-processing" Nov 03 23:05:40.539638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539553 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fbef3b-965d-4a61-bebc-63f605f0b47f" containerName="payload-pre-processing" Nov 03 23:05:40.539638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539572 5272 cpu_manager.go:414] "containerMap: removing container" podUID="95901769-38a7-4687-ae42-79dc16ab6c6a" containerName="payload-processing" Nov 03 23:05:40.539638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539576 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95901769-38a7-4687-ae42-79dc16ab6c6a" containerName="payload-processing" Nov 03 23:05:40.539766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539641 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" Nov 03 23:05:40.539766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539654 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fbef3b-965d-4a61-bebc-63f605f0b47f" containerName="payload-pre-processing" Nov 03 23:05:40.539766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539661 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" Nov 03 23:05:40.539766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539668 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="95901769-38a7-4687-ae42-79dc16ab6c6a" containerName="payload-processing" Nov 03 23:05:40.539766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539674 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="71772ded-d58f-4de6-b306-64bc73c833f7" containerName="revoke-keys" Nov 03 23:05:40.539766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.539682 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a587b1b-f8dd-41f3-9766-cc48b39a60a9" containerName="istio-proxy" Nov 03 23:05:40.543000 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.542979 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.592116 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.592085 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ac6633\"/\"test-model-ac6633-kserve-self-signed-certs\"" Nov 03 23:05:40.638699 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.638666 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ac6633\"/\"openshift-service-ca.crt\"" Nov 03 23:05:40.638699 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.638665 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ac6633\"/\"kube-root-ca.crt\"" Nov 03 23:05:40.641472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.641446 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq"] Nov 03 23:05:40.645063 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.645044 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1347c693-c8d5-4f4a-8408-9250ef8619c7-tls-certs\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.645126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.645098 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-tmp-dir\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.645188 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.645168 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-model-cache\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.645293 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.645271 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-home\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.645366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.645315 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-dshm\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.645431 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.645375 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwtw7\" (UniqueName: \"kubernetes.io/projected/1347c693-c8d5-4f4a-8408-9250ef8619c7-kube-api-access-dwtw7\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.696829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.693063 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ac6633\"/\"default-dockercfg-dr6q6\"" Nov 03 23:05:40.746539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.746498 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-home\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.746539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.746540 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-dshm\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.746775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.746566 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwtw7\" (UniqueName: \"kubernetes.io/projected/1347c693-c8d5-4f4a-8408-9250ef8619c7-kube-api-access-dwtw7\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.746775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.746636 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1347c693-c8d5-4f4a-8408-9250ef8619c7-tls-certs\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.746775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.746720 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-tmp-dir\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.746775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.746746 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-model-cache\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.747056 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.746998 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-home\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.747151 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.747110 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-tmp-dir\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.747217 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.747163 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-model-cache\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.749017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.748992 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-dshm\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.749227 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.749213 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1347c693-c8d5-4f4a-8408-9250ef8619c7-tls-certs\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.806640 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.806576 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwtw7\" (UniqueName: \"kubernetes.io/projected/1347c693-c8d5-4f4a-8408-9250ef8619c7-kube-api-access-dwtw7\") pod \"test-model-ac6633-kserve-869955d667-zsgcq\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:40.856467 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:40.856436 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:41.236200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:41.236123 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq"] Nov 03 23:05:41.242482 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:05:41.242457 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1347c693_c8d5_4f4a_8408_9250ef8619c7.slice/crio-fcd8e1fdfec62882a93901ea41f70b3a926f2b187dbb641aa1e28b89dc4ab3de WatchSource:0}: Error finding container fcd8e1fdfec62882a93901ea41f70b3a926f2b187dbb641aa1e28b89dc4ab3de: Status 404 returned error can't find the container with id fcd8e1fdfec62882a93901ea41f70b3a926f2b187dbb641aa1e28b89dc4ab3de Nov 03 23:05:41.385460 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:41.385423 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" event={"ID":"1347c693-c8d5-4f4a-8408-9250ef8619c7","Type":"ContainerStarted","Data":"c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190"} Nov 03 23:05:41.385633 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:41.385473 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" event={"ID":"1347c693-c8d5-4f4a-8408-9250ef8619c7","Type":"ContainerStarted","Data":"fcd8e1fdfec62882a93901ea41f70b3a926f2b187dbb641aa1e28b89dc4ab3de"} Nov 03 23:05:41.495013 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:41.494914 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" podStartSLOduration=1.494901254 podStartE2EDuration="1.494901254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:41.494261347 +0000 UTC m=+2522.116569658" watchObservedRunningTime="2026-11-03 23:05:41.494901254 +0000 UTC m=+2522.117209627" Nov 03 23:05:41.857382 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:41.857335 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:42.412983 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:42.412940 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:05:43.620337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:43.619609 5272 scope.go:122] "RemoveContainer" containerID="dcbad510245848ec1a09efbb4f50e26383f6e729bea8867d1c34f74f56874a6c" Nov 03 23:05:50.857643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:50.857553 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:50.870291 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:50.870272 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:51.464297 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:51.464265 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:05:52.936820 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.936776 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s"] Nov 03 23:05:52.937289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.937267 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" Nov 03 23:05:52.937289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.937283 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9060091-788b-47ed-8934-18fc8c9bc462" containerName="istio-proxy" Nov 03 23:05:52.942360 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.942341 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:52.991565 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.991539 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:52.992135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.992112 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:05:52.992135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.992131 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:05:52.992332 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.992177 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:05:52.995768 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:52.995745 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s"] Nov 03 23:05:53.075539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.075507 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.075659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.075545 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.075659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.075616 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn7d7\" (UniqueName: \"kubernetes.io/projected/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-kube-api-access-fn7d7\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.176883 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.176862 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.176890 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.177004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.176923 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7d7\" (UniqueName: \"kubernetes.io/projected/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-kube-api-access-fn7d7\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.177499 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.177468 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.179200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.179181 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.234613 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.234557 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7d7\" (UniqueName: \"kubernetes.io/projected/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-kube-api-access-fn7d7\") pod \"maas-api-revoke-sub-28692ad6-67x9s\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.255760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.255738 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:53.647543 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:53.647510 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s"] Nov 03 23:05:53.653856 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:05:53.653827 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627d1908_fcf8_4d9d_bb70_22fbf83f8d7e.slice/crio-a61f8b813894eb8a3567654380847ad64a676479a5e947c5a3d0c743ffe2506e WatchSource:0}: Error finding container a61f8b813894eb8a3567654380847ad64a676479a5e947c5a3d0c743ffe2506e: Status 404 returned error can't find the container with id a61f8b813894eb8a3567654380847ad64a676479a5e947c5a3d0c743ffe2506e Nov 03 23:05:54.475917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:54.475867 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" event={"ID":"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e","Type":"ContainerStarted","Data":"707f66bcf62de45185eeb8e333967ba08294a3aaeab1abf37d0be53e89dce6fd"} Nov 03 23:05:54.476435 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:54.475936 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" event={"ID":"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e","Type":"ContainerStarted","Data":"a61f8b813894eb8a3567654380847ad64a676479a5e947c5a3d0c743ffe2506e"} Nov 03 23:05:54.500875 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:05:54.500842 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:05:54.586146 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:54.586091 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" podStartSLOduration=2.586071716 podStartE2EDuration="2.586071716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:54.584823141 +0000 UTC m=+2535.207131475" watchObservedRunningTime="2026-11-03 23:05:54.586071716 +0000 UTC m=+2535.208380023" Nov 03 23:05:55.483971 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:55.483935 5272 generic.go:356] "Generic (PLEG): container finished" podID="627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" containerID="707f66bcf62de45185eeb8e333967ba08294a3aaeab1abf37d0be53e89dce6fd" exitCode=0 Nov 03 23:05:55.484405 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:55.484022 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" event={"ID":"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e","Type":"ContainerDied","Data":"707f66bcf62de45185eeb8e333967ba08294a3aaeab1abf37d0be53e89dce6fd"} Nov 03 23:05:56.646708 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.646683 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:56.817258 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.817230 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-kube-api-access-fn7d7\" (UniqueName: \"kubernetes.io/projected/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-kube-api-access-fn7d7\") pod \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " Nov 03 23:05:56.817415 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.817373 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-openshift-ca\") pod \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " Nov 03 23:05:56.817415 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.817403 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-xks-ca\") pod \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\" (UID: \"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e\") " Nov 03 23:05:56.817819 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.817776 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-openshift-ca" pod "627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" (UID: "627d1908-fcf8-4d9d-bb70-22fbf83f8d7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:56.819309 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.819283 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-kube-api-access-fn7d7" pod "627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" (UID: "627d1908-fcf8-4d9d-bb70-22fbf83f8d7e"). InnerVolumeSpecName "kube-api-access-fn7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:56.819453 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.819424 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-xks-ca" pod "627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" (UID: "627d1908-fcf8-4d9d-bb70-22fbf83f8d7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:56.918713 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.918685 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:56.918713 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.918713 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:56.918918 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:56.918723 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn7d7\" (UniqueName: \"kubernetes.io/projected/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e-kube-api-access-fn7d7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:05:57.503239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:57.503209 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" event={"ID":"627d1908-fcf8-4d9d-bb70-22fbf83f8d7e","Type":"ContainerDied","Data":"a61f8b813894eb8a3567654380847ad64a676479a5e947c5a3d0c743ffe2506e"} Nov 03 23:05:57.503239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:57.503235 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s" Nov 03 23:05:57.503407 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:05:57.503261 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61f8b813894eb8a3567654380847ad64a676479a5e947c5a3d0c743ffe2506e" Nov 03 23:06:04.393939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:04.393905 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj"] Nov 03 23:06:04.440047 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:04.440021 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-656ee6-5jrcj"] Nov 03 23:06:04.940921 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:04.940890 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5"] Nov 03 23:06:05.492440 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:05.492408 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71772ded-d58f-4de6-b306-64bc73c833f7" path="/var/lib/kubelet/pods/71772ded-d58f-4de6-b306-64bc73c833f7/volumes" Nov 03 23:06:06.155544 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.155509 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8"] Nov 03 23:06:06.156119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.156101 5272 cpu_manager.go:414] "containerMap: removing container" podUID="627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" containerName="revoke-keys" Nov 03 23:06:06.156201 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.156120 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" containerName="revoke-keys" Nov 03 23:06:06.156242 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.156237 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" containerName="revoke-keys" Nov 03 23:06:06.159659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.159639 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.220149 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.220119 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8"] Nov 03 23:06:06.228736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.228711 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.228900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.228754 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2w6f\" (UniqueName: \"kubernetes.io/projected/ecf38b79-e49e-4974-a43e-809ea6bed7cf-kube-api-access-r2w6f\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.228977 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.228954 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.263482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.263460 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:06:06.263482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.263460 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:06.263643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.263484 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:06.263643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.263460 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:06.330506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.330480 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.330628 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.330520 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2w6f\" (UniqueName: \"kubernetes.io/projected/ecf38b79-e49e-4974-a43e-809ea6bed7cf-kube-api-access-r2w6f\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.330628 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.330584 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.331224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.331202 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.332834 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.332795 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.388081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.388053 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2w6f\" (UniqueName: \"kubernetes.io/projected/ecf38b79-e49e-4974-a43e-809ea6bed7cf-kube-api-access-r2w6f\") pod \"maas-api-revoke-sub-78e05e65-gdrq8\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.472381 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.472325 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:06.844488 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:06.844452 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8"] Nov 03 23:06:06.850672 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:06:06.850646 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf38b79_e49e_4974_a43e_809ea6bed7cf.slice/crio-85ede3e31f8cb41285d836d7a79f6da3b479ab7517b30c41c751549821706f38 WatchSource:0}: Error finding container 85ede3e31f8cb41285d836d7a79f6da3b479ab7517b30c41c751549821706f38: Status 404 returned error can't find the container with id 85ede3e31f8cb41285d836d7a79f6da3b479ab7517b30c41c751549821706f38 Nov 03 23:06:07.569093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:07.569058 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" event={"ID":"ecf38b79-e49e-4974-a43e-809ea6bed7cf","Type":"ContainerStarted","Data":"c6bca53a44bed1b600386e778ecb3c6b9a9ffb8f753461e301a447e1e5f02888"} Nov 03 23:06:07.569561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:07.569115 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" event={"ID":"ecf38b79-e49e-4974-a43e-809ea6bed7cf","Type":"ContainerStarted","Data":"85ede3e31f8cb41285d836d7a79f6da3b479ab7517b30c41c751549821706f38"} Nov 03 23:06:07.723051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:07.723005 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" podStartSLOduration=1.722992656 podStartE2EDuration="1.722992656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:07.676513522 +0000 UTC m=+2548.298821828" watchObservedRunningTime="2026-11-03 23:06:07.722992656 +0000 UTC m=+2548.345300962" Nov 03 23:06:08.577468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:08.577433 5272 generic.go:356] "Generic (PLEG): container finished" podID="ecf38b79-e49e-4974-a43e-809ea6bed7cf" containerID="c6bca53a44bed1b600386e778ecb3c6b9a9ffb8f753461e301a447e1e5f02888" exitCode=0 Nov 03 23:06:08.577903 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:08.577529 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" event={"ID":"ecf38b79-e49e-4974-a43e-809ea6bed7cf","Type":"ContainerDied","Data":"c6bca53a44bed1b600386e778ecb3c6b9a9ffb8f753461e301a447e1e5f02888"} Nov 03 23:06:09.721655 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.721621 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:09.769824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.769244 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-openshift-ca\") pod \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " Nov 03 23:06:09.769824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.769339 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf38b79-e49e-4974-a43e-809ea6bed7cf-kube-api-access-r2w6f\" (UniqueName: \"kubernetes.io/projected/ecf38b79-e49e-4974-a43e-809ea6bed7cf-kube-api-access-r2w6f\") pod \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " Nov 03 23:06:09.769824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.769452 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-xks-ca\") pod \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\" (UID: \"ecf38b79-e49e-4974-a43e-809ea6bed7cf\") " Nov 03 23:06:09.784842 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.770426 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-openshift-ca" pod "ecf38b79-e49e-4974-a43e-809ea6bed7cf" (UID: "ecf38b79-e49e-4974-a43e-809ea6bed7cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:09.784842 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.782972 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf38b79-e49e-4974-a43e-809ea6bed7cf-kube-api-access-r2w6f" pod "ecf38b79-e49e-4974-a43e-809ea6bed7cf" (UID: "ecf38b79-e49e-4974-a43e-809ea6bed7cf"). InnerVolumeSpecName "kube-api-access-r2w6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:09.784842 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.783193 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.784842 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.783219 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2w6f\" (UniqueName: \"kubernetes.io/projected/ecf38b79-e49e-4974-a43e-809ea6bed7cf-kube-api-access-r2w6f\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.788164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.787928 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-xks-ca" pod "ecf38b79-e49e-4974-a43e-809ea6bed7cf" (UID: "ecf38b79-e49e-4974-a43e-809ea6bed7cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:09.884260 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:09.884190 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf38b79-e49e-4974-a43e-809ea6bed7cf-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:10.069992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:10.069953 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:06:10.608740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:10.608687 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" event={"ID":"ecf38b79-e49e-4974-a43e-809ea6bed7cf","Type":"ContainerDied","Data":"85ede3e31f8cb41285d836d7a79f6da3b479ab7517b30c41c751549821706f38"} Nov 03 23:06:10.609010 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:10.608989 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85ede3e31f8cb41285d836d7a79f6da3b479ab7517b30c41c751549821706f38" Nov 03 23:06:10.609199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:10.609185 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8" Nov 03 23:06:12.753744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.753706 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn"] Nov 03 23:06:12.754310 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.754290 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ecf38b79-e49e-4974-a43e-809ea6bed7cf" containerName="revoke-keys" Nov 03 23:06:12.754310 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.754310 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf38b79-e49e-4974-a43e-809ea6bed7cf" containerName="revoke-keys" Nov 03 23:06:12.754472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.754438 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf38b79-e49e-4974-a43e-809ea6bed7cf" containerName="revoke-keys" Nov 03 23:06:12.758296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.758277 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.812000 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.811957 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn"] Nov 03 23:06:12.831018 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.830982 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.831291 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.831263 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.831369 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.831350 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt22f\" (UniqueName: \"kubernetes.io/projected/cb570a4f-29de-4fab-adde-bdea617245c7-kube-api-access-pt22f\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.854811 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.854784 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:06:12.854811 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.854794 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:12.855059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.854820 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:12.855059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.854846 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:12.932526 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.932500 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.932688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.932553 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt22f\" (UniqueName: \"kubernetes.io/projected/cb570a4f-29de-4fab-adde-bdea617245c7-kube-api-access-pt22f\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.932688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.932586 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.933328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.933300 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:12.934971 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:12.934948 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:13.011236 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:13.011175 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt22f\" (UniqueName: \"kubernetes.io/projected/cb570a4f-29de-4fab-adde-bdea617245c7-kube-api-access-pt22f\") pod \"maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:13.076081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:13.076053 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:13.454001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:13.453974 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn"] Nov 03 23:06:13.461442 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:06:13.461401 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb570a4f_29de_4fab_adde_bdea617245c7.slice/crio-41a4eeb7b3c1c9ce359621cc63e3b4d1e69be7af5d4595f385adbd96d7027e3d WatchSource:0}: Error finding container 41a4eeb7b3c1c9ce359621cc63e3b4d1e69be7af5d4595f385adbd96d7027e3d: Status 404 returned error can't find the container with id 41a4eeb7b3c1c9ce359621cc63e3b4d1e69be7af5d4595f385adbd96d7027e3d Nov 03 23:06:13.630571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:13.630540 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" event={"ID":"cb570a4f-29de-4fab-adde-bdea617245c7","Type":"ContainerStarted","Data":"41a4eeb7b3c1c9ce359621cc63e3b4d1e69be7af5d4595f385adbd96d7027e3d"} Nov 03 23:06:14.638894 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:14.638797 5272 generic.go:356] "Generic (PLEG): container finished" podID="cb570a4f-29de-4fab-adde-bdea617245c7" containerID="41df2883e791308fa227d82031d3e4731a0ccb7e3bfc5efd65a29b1b21321fd5" exitCode=0 Nov 03 23:06:14.639255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:14.638888 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" event={"ID":"cb570a4f-29de-4fab-adde-bdea617245c7","Type":"ContainerDied","Data":"41df2883e791308fa227d82031d3e4731a0ccb7e3bfc5efd65a29b1b21321fd5"} Nov 03 23:06:15.784931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.784907 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:15.864164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.864138 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-openshift-ca\") pod \"cb570a4f-29de-4fab-adde-bdea617245c7\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " Nov 03 23:06:15.864293 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.864278 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-xks-ca\") pod \"cb570a4f-29de-4fab-adde-bdea617245c7\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " Nov 03 23:06:15.864371 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.864357 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb570a4f-29de-4fab-adde-bdea617245c7-kube-api-access-pt22f\" (UniqueName: \"kubernetes.io/projected/cb570a4f-29de-4fab-adde-bdea617245c7-kube-api-access-pt22f\") pod \"cb570a4f-29de-4fab-adde-bdea617245c7\" (UID: \"cb570a4f-29de-4fab-adde-bdea617245c7\") " Nov 03 23:06:15.864474 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.864453 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-openshift-ca" pod "cb570a4f-29de-4fab-adde-bdea617245c7" (UID: "cb570a4f-29de-4fab-adde-bdea617245c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:15.864643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.864627 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:15.866228 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.866206 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-xks-ca" pod "cb570a4f-29de-4fab-adde-bdea617245c7" (UID: "cb570a4f-29de-4fab-adde-bdea617245c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:15.866345 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.866324 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb570a4f-29de-4fab-adde-bdea617245c7-kube-api-access-pt22f" pod "cb570a4f-29de-4fab-adde-bdea617245c7" (UID: "cb570a4f-29de-4fab-adde-bdea617245c7"). InnerVolumeSpecName "kube-api-access-pt22f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:15.965866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.965772 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb570a4f-29de-4fab-adde-bdea617245c7-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:15.965866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:15.965824 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt22f\" (UniqueName: \"kubernetes.io/projected/cb570a4f-29de-4fab-adde-bdea617245c7-kube-api-access-pt22f\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:16.652257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:16.652231 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" Nov 03 23:06:16.652257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:16.652243 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn" event={"ID":"cb570a4f-29de-4fab-adde-bdea617245c7","Type":"ContainerDied","Data":"41a4eeb7b3c1c9ce359621cc63e3b4d1e69be7af5d4595f385adbd96d7027e3d"} Nov 03 23:06:16.652488 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:16.652296 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41a4eeb7b3c1c9ce359621cc63e3b4d1e69be7af5d4595f385adbd96d7027e3d" Nov 03 23:06:17.184424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:17.184389 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv"] Nov 03 23:06:17.230547 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:17.230518 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw"] Nov 03 23:06:19.960641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:19.960599 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" podUID="7677b675-ffb7-4481-beec-5a7b78fae29b" containerName="main" containerID="cri-o://c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b" gracePeriod=45 Nov 03 23:06:20.215599 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.215536 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:06:20.308461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308426 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7677b675-ffb7-4481-beec-5a7b78fae29b-kube-api-access-whrwq\" (UniqueName: \"kubernetes.io/projected/7677b675-ffb7-4481-beec-5a7b78fae29b-kube-api-access-whrwq\") pod \"7677b675-ffb7-4481-beec-5a7b78fae29b\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " Nov 03 23:06:20.308659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308497 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7677b675-ffb7-4481-beec-5a7b78fae29b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7677b675-ffb7-4481-beec-5a7b78fae29b-tls-certs\") pod \"7677b675-ffb7-4481-beec-5a7b78fae29b\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " Nov 03 23:06:20.308659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308554 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-home\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-home\") pod \"7677b675-ffb7-4481-beec-5a7b78fae29b\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " Nov 03 23:06:20.308659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308596 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-tmp-dir\") pod \"7677b675-ffb7-4481-beec-5a7b78fae29b\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " Nov 03 23:06:20.308659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308630 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-dshm\") pod \"7677b675-ffb7-4481-beec-5a7b78fae29b\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " Nov 03 23:06:20.308950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308725 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-model-cache\") pod \"7677b675-ffb7-4481-beec-5a7b78fae29b\" (UID: \"7677b675-ffb7-4481-beec-5a7b78fae29b\") " Nov 03 23:06:20.308950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308840 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-tmp-dir" pod "7677b675-ffb7-4481-beec-5a7b78fae29b" (UID: "7677b675-ffb7-4481-beec-5a7b78fae29b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:20.308950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.308856 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-home" pod "7677b675-ffb7-4481-beec-5a7b78fae29b" (UID: "7677b675-ffb7-4481-beec-5a7b78fae29b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:20.309115 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.309002 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-model-cache" pod "7677b675-ffb7-4481-beec-5a7b78fae29b" (UID: "7677b675-ffb7-4481-beec-5a7b78fae29b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:20.309174 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.309162 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:20.309232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.309175 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:20.309232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.309202 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:20.310888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.310857 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-dshm" pod "7677b675-ffb7-4481-beec-5a7b78fae29b" (UID: "7677b675-ffb7-4481-beec-5a7b78fae29b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:20.310888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.310876 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7677b675-ffb7-4481-beec-5a7b78fae29b-tls-certs" pod "7677b675-ffb7-4481-beec-5a7b78fae29b" (UID: "7677b675-ffb7-4481-beec-5a7b78fae29b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:20.311040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.310950 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7677b675-ffb7-4481-beec-5a7b78fae29b-kube-api-access-whrwq" pod "7677b675-ffb7-4481-beec-5a7b78fae29b" (UID: "7677b675-ffb7-4481-beec-5a7b78fae29b"). InnerVolumeSpecName "kube-api-access-whrwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:20.410765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.410730 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7677b675-ffb7-4481-beec-5a7b78fae29b-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:20.410765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.410759 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrwq\" (UniqueName: \"kubernetes.io/projected/7677b675-ffb7-4481-beec-5a7b78fae29b-kube-api-access-whrwq\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:20.410765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.410771 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7677b675-ffb7-4481-beec-5a7b78fae29b-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:20.679570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.679533 5272 generic.go:356] "Generic (PLEG): container finished" podID="7677b675-ffb7-4481-beec-5a7b78fae29b" containerID="c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b" exitCode=0 Nov 03 23:06:20.679759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.679577 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" Nov 03 23:06:20.679759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.679620 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" event={"ID":"7677b675-ffb7-4481-beec-5a7b78fae29b","Type":"ContainerDied","Data":"c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b"} Nov 03 23:06:20.679759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.679664 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5" event={"ID":"7677b675-ffb7-4481-beec-5a7b78fae29b","Type":"ContainerDied","Data":"30bdaac8d43dbe52eba2b8fe4e2b3f5751dd119fdb9d7e6b22950d04c4028337"} Nov 03 23:06:20.679759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.679685 5272 scope.go:122] "RemoveContainer" containerID="c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b" Nov 03 23:06:20.690073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.690056 5272 scope.go:122] "RemoveContainer" containerID="c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b" Nov 03 23:06:20.690358 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:06:20.690340 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b\": container with ID starting with c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b not found: ID does not exist" containerID="c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b" Nov 03 23:06:20.690407 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.690368 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b"} err="failed to get container status \"c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b\": rpc error: code = NotFound desc = could not find container \"c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b\": container with ID starting with c49fe8459d1dc10df4296c76aa1fa14fed3bddde62ba3f40962a464d3c03ab4b not found: ID does not exist" Nov 03 23:06:20.833104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.833076 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5"] Nov 03 23:06:20.840031 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:20.840007 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9b4383/test-model-9b4383-kserve-796d57944f-k67d5"] Nov 03 23:06:21.493558 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:21.493521 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7677b675-ffb7-4481-beec-5a7b78fae29b" path="/var/lib/kubelet/pods/7677b675-ffb7-4481-beec-5a7b78fae29b/volumes" Nov 03 23:06:24.265990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:24.265954 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh"] Nov 03 23:06:24.375109 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:24.375073 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s"] Nov 03 23:06:24.517245 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:06:24.517139 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:06:26.044778 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:26.044741 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4"] Nov 03 23:06:26.092266 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:26.092228 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd1d6fc-9qlh4"] Nov 03 23:06:27.044546 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:27.044498 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq"] Nov 03 23:06:27.091184 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:27.091154 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56b8d6a7-bscmq"] Nov 03 23:06:27.494135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:27.494059 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06db652e-54ce-4405-b7af-fdcf65b8bf0b" path="/var/lib/kubelet/pods/06db652e-54ce-4405-b7af-fdcf65b8bf0b/volumes" Nov 03 23:06:27.494495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:27.494481 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913fcaca-efec-4e75-a731-5aade94d4831" path="/var/lib/kubelet/pods/913fcaca-efec-4e75-a731-5aade94d4831/volumes" Nov 03 23:06:29.266285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.266248 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" podUID="1e06ab06-701a-4213-b11e-3e4be2f716af" containerName="payload-processing" containerID="cri-o://6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022" gracePeriod=25 Nov 03 23:06:29.376118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.376083 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" podUID="5bf81353-20bd-47f2-9dd1-cf95f768a565" containerName="payload-pre-processing" containerID="cri-o://0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382" gracePeriod=25 Nov 03 23:06:29.542742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.542714 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 23:06:29.647001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.646976 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 23:06:29.718102 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.718073 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e06ab06-701a-4213-b11e-3e4be2f716af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e06ab06-701a-4213-b11e-3e4be2f716af-plugins-config-volume\") pod \"1e06ab06-701a-4213-b11e-3e4be2f716af\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " Nov 03 23:06:29.718271 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.718133 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e06ab06-701a-4213-b11e-3e4be2f716af-kube-api-access-t2nls\" (UniqueName: \"kubernetes.io/projected/1e06ab06-701a-4213-b11e-3e4be2f716af-kube-api-access-t2nls\") pod \"1e06ab06-701a-4213-b11e-3e4be2f716af\" (UID: \"1e06ab06-701a-4213-b11e-3e4be2f716af\") " Nov 03 23:06:29.718444 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.718423 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e06ab06-701a-4213-b11e-3e4be2f716af-plugins-config-volume" pod "1e06ab06-701a-4213-b11e-3e4be2f716af" (UID: "1e06ab06-701a-4213-b11e-3e4be2f716af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:29.720231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.720212 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e06ab06-701a-4213-b11e-3e4be2f716af-kube-api-access-t2nls" pod "1e06ab06-701a-4213-b11e-3e4be2f716af" (UID: "1e06ab06-701a-4213-b11e-3e4be2f716af"). InnerVolumeSpecName "kube-api-access-t2nls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:29.787110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.787083 5272 generic.go:356] "Generic (PLEG): container finished" podID="5bf81353-20bd-47f2-9dd1-cf95f768a565" containerID="0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382" exitCode=0 Nov 03 23:06:29.787296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.787117 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" event={"ID":"5bf81353-20bd-47f2-9dd1-cf95f768a565","Type":"ContainerDied","Data":"0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382"} Nov 03 23:06:29.787296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.787146 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" Nov 03 23:06:29.787296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.787184 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s" event={"ID":"5bf81353-20bd-47f2-9dd1-cf95f768a565","Type":"ContainerDied","Data":"13d7fda8b8839a386440334d7882d36c32d2ed3546633dfe078245850b5a7a23"} Nov 03 23:06:29.787296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.787216 5272 scope.go:122] "RemoveContainer" containerID="0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382" Nov 03 23:06:29.789250 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.789226 5272 generic.go:356] "Generic (PLEG): container finished" podID="1e06ab06-701a-4213-b11e-3e4be2f716af" containerID="6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022" exitCode=0 Nov 03 23:06:29.789556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.789273 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" Nov 03 23:06:29.789556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.789373 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" event={"ID":"1e06ab06-701a-4213-b11e-3e4be2f716af","Type":"ContainerDied","Data":"6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022"} Nov 03 23:06:29.789556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.789417 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh" event={"ID":"1e06ab06-701a-4213-b11e-3e4be2f716af","Type":"ContainerDied","Data":"cb61053c00c08f930b949443e148c476eb9f3b580f9344e81dc88ba4bedaac88"} Nov 03 23:06:29.800272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.800249 5272 scope.go:122] "RemoveContainer" containerID="0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382" Nov 03 23:06:29.800547 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:06:29.800525 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382\": container with ID starting with 0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382 not found: ID does not exist" containerID="0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382" Nov 03 23:06:29.800622 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.800557 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382"} err="failed to get container status \"0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382\": rpc error: code = NotFound desc = could not find container \"0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382\": container with ID starting with 0e43b1fe49f524a7ddefd5a28b3b7ce6bb8130d981574fe030493a1cb03de382 not found: ID does not exist" Nov 03 23:06:29.800622 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.800581 5272 scope.go:122] "RemoveContainer" containerID="6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022" Nov 03 23:06:29.809104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.809085 5272 scope.go:122] "RemoveContainer" containerID="6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022" Nov 03 23:06:29.809360 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:06:29.809342 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022\": container with ID starting with 6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022 not found: ID does not exist" containerID="6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022" Nov 03 23:06:29.809425 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.809371 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022"} err="failed to get container status \"6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022\": rpc error: code = NotFound desc = could not find container \"6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022\": container with ID starting with 6fef84ac213ed85c683c04a674e2c8d4753d160d039a282569d82f4a1ea86022 not found: ID does not exist" Nov 03 23:06:29.819750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.819730 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bf81353-20bd-47f2-9dd1-cf95f768a565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bf81353-20bd-47f2-9dd1-cf95f768a565-plugins-config-volume\") pod \"5bf81353-20bd-47f2-9dd1-cf95f768a565\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " Nov 03 23:06:29.819852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.819765 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf81353-20bd-47f2-9dd1-cf95f768a565-kube-api-access-qkpwp\" (UniqueName: \"kubernetes.io/projected/5bf81353-20bd-47f2-9dd1-cf95f768a565-kube-api-access-qkpwp\") pod \"5bf81353-20bd-47f2-9dd1-cf95f768a565\" (UID: \"5bf81353-20bd-47f2-9dd1-cf95f768a565\") " Nov 03 23:06:29.820130 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.820115 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e06ab06-701a-4213-b11e-3e4be2f716af-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:29.820171 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.820133 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2nls\" (UniqueName: \"kubernetes.io/projected/1e06ab06-701a-4213-b11e-3e4be2f716af-kube-api-access-t2nls\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:29.820171 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.820131 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf81353-20bd-47f2-9dd1-cf95f768a565-plugins-config-volume" pod "5bf81353-20bd-47f2-9dd1-cf95f768a565" (UID: "5bf81353-20bd-47f2-9dd1-cf95f768a565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:29.821665 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.821647 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf81353-20bd-47f2-9dd1-cf95f768a565-kube-api-access-qkpwp" pod "5bf81353-20bd-47f2-9dd1-cf95f768a565" (UID: "5bf81353-20bd-47f2-9dd1-cf95f768a565"). InnerVolumeSpecName "kube-api-access-qkpwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:29.921380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.921351 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bf81353-20bd-47f2-9dd1-cf95f768a565-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:29.921380 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.921382 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpwp\" (UniqueName: \"kubernetes.io/projected/5bf81353-20bd-47f2-9dd1-cf95f768a565-kube-api-access-qkpwp\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:29.944300 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.944272 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh"] Nov 03 23:06:29.957853 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:29.951692 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5e72ad-69d4b7cbb-r4xrh"] Nov 03 23:06:30.204946 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.204914 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s"] Nov 03 23:06:30.209392 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.209362 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5e72ad-65d5d94fc9-6r57s"] Nov 03 23:06:30.708190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708155 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2"] Nov 03 23:06:30.708732 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708716 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5bf81353-20bd-47f2-9dd1-cf95f768a565" containerName="payload-pre-processing" Nov 03 23:06:30.708787 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708734 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf81353-20bd-47f2-9dd1-cf95f768a565" containerName="payload-pre-processing" Nov 03 23:06:30.708787 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708754 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7677b675-ffb7-4481-beec-5a7b78fae29b" containerName="main" Nov 03 23:06:30.708787 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708759 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7677b675-ffb7-4481-beec-5a7b78fae29b" containerName="main" Nov 03 23:06:30.708787 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708781 5272 cpu_manager.go:414] "containerMap: removing container" podUID="cb570a4f-29de-4fab-adde-bdea617245c7" containerName="revoke-keys" Nov 03 23:06:30.708787 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708786 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb570a4f-29de-4fab-adde-bdea617245c7" containerName="revoke-keys" Nov 03 23:06:30.708988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708824 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1e06ab06-701a-4213-b11e-3e4be2f716af" containerName="payload-processing" Nov 03 23:06:30.708988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708830 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e06ab06-701a-4213-b11e-3e4be2f716af" containerName="payload-processing" Nov 03 23:06:30.708988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708917 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e06ab06-701a-4213-b11e-3e4be2f716af" containerName="payload-processing" Nov 03 23:06:30.708988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708928 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf81353-20bd-47f2-9dd1-cf95f768a565" containerName="payload-pre-processing" Nov 03 23:06:30.708988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708937 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7677b675-ffb7-4481-beec-5a7b78fae29b" containerName="main" Nov 03 23:06:30.708988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.708949 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb570a4f-29de-4fab-adde-bdea617245c7" containerName="revoke-keys" Nov 03 23:06:30.713859 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.713839 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.763444 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.763418 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:30.763613 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.763593 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:30.763702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.763642 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:30.763702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.763656 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:06:30.812327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.812295 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2"] Nov 03 23:06:30.831677 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.831642 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.831833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.831781 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.831894 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.831880 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpxf\" (UniqueName: \"kubernetes.io/projected/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-kube-api-access-dxpxf\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.932501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.932469 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.932657 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.932551 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.932657 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.932604 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpxf\" (UniqueName: \"kubernetes.io/projected/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-kube-api-access-dxpxf\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.933259 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.933233 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.934862 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.934841 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:30.991010 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:30.990945 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpxf\" (UniqueName: \"kubernetes.io/projected/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-kube-api-access-dxpxf\") pod \"maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:31.027905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:31.027878 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:31.403060 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:31.403020 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2"] Nov 03 23:06:31.409621 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:06:31.409591 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0777fdd_20ab_471d_9a37_42a03e4d9ddd.slice/crio-54a089155d242955978ef9afd2df99631a7123eea9bfa8bafcb17b7be5521464 WatchSource:0}: Error finding container 54a089155d242955978ef9afd2df99631a7123eea9bfa8bafcb17b7be5521464: Status 404 returned error can't find the container with id 54a089155d242955978ef9afd2df99631a7123eea9bfa8bafcb17b7be5521464 Nov 03 23:06:31.493252 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:31.493216 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e06ab06-701a-4213-b11e-3e4be2f716af" path="/var/lib/kubelet/pods/1e06ab06-701a-4213-b11e-3e4be2f716af/volumes" Nov 03 23:06:31.493628 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:31.493614 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf81353-20bd-47f2-9dd1-cf95f768a565" path="/var/lib/kubelet/pods/5bf81353-20bd-47f2-9dd1-cf95f768a565/volumes" Nov 03 23:06:31.807931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:31.807893 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" event={"ID":"d0777fdd-20ab-471d-9a37-42a03e4d9ddd","Type":"ContainerStarted","Data":"8c9f87c4335f19bc6896ed38cda51a843f13b7133645107044a704d0abb93cf6"} Nov 03 23:06:31.808404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:31.807947 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" event={"ID":"d0777fdd-20ab-471d-9a37-42a03e4d9ddd","Type":"ContainerStarted","Data":"54a089155d242955978ef9afd2df99631a7123eea9bfa8bafcb17b7be5521464"} Nov 03 23:06:31.923107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:31.920067 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" podStartSLOduration=1.920048411 podStartE2EDuration="1.920048411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:31.918200568 +0000 UTC m=+2572.540508877" watchObservedRunningTime="2026-11-03 23:06:31.920048411 +0000 UTC m=+2572.542356723" Nov 03 23:06:32.205141 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.205046 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" containerID="cri-o://7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989" gracePeriod=45 Nov 03 23:06:32.250081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.250036 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" containerID="cri-o://bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9" gracePeriod=45 Nov 03 23:06:32.254757 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.254727 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": EOF" Nov 03 23:06:32.546138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.546115 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 23:06:32.550014 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.549995 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 23:06:32.649112 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649082 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-model-cache\") pod \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " Nov 03 23:06:32.649112 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649122 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-dshm\") pod \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " Nov 03 23:06:32.649394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649163 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-tmp-dir\") pod \"f3b720de-005d-4af7-b356-2d2387bf4e57\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " Nov 03 23:06:32.649394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649202 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b720de-005d-4af7-b356-2d2387bf4e57-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b720de-005d-4af7-b356-2d2387bf4e57-tls-certs\") pod \"f3b720de-005d-4af7-b356-2d2387bf4e57\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " Nov 03 23:06:32.649394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649238 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tls-certs\") pod \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " Nov 03 23:06:32.649394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649271 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-home\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-home\") pod \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " Nov 03 23:06:32.649394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649310 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tmp-dir\") pod \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " Nov 03 23:06:32.649394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649359 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-home\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-home\") pod \"f3b720de-005d-4af7-b356-2d2387bf4e57\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649399 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b720de-005d-4af7-b356-2d2387bf4e57-kube-api-access-22p7k\" (UniqueName: \"kubernetes.io/projected/f3b720de-005d-4af7-b356-2d2387bf4e57-kube-api-access-22p7k\") pod \"f3b720de-005d-4af7-b356-2d2387bf4e57\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649415 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-model-cache" pod "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" (UID: "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649429 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-tmp-dir" pod "f3b720de-005d-4af7-b356-2d2387bf4e57" (UID: "f3b720de-005d-4af7-b356-2d2387bf4e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649470 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-kube-api-access-f7jmk\" (UniqueName: \"kubernetes.io/projected/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-kube-api-access-f7jmk\") pod \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\" (UID: \"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de\") " Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649526 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-model-cache\") pod \"f3b720de-005d-4af7-b356-2d2387bf4e57\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649611 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-home" pod "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" (UID: "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649636 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tmp-dir" pod "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" (UID: "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.649734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649643 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-dshm\") pod \"f3b720de-005d-4af7-b356-2d2387bf4e57\" (UID: \"f3b720de-005d-4af7-b356-2d2387bf4e57\") " Nov 03 23:06:32.650208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.649751 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-home" pod "f3b720de-005d-4af7-b356-2d2387bf4e57" (UID: "f3b720de-005d-4af7-b356-2d2387bf4e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.650272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.650228 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-model-cache" pod "f3b720de-005d-4af7-b356-2d2387bf4e57" (UID: "f3b720de-005d-4af7-b356-2d2387bf4e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.650338 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.650320 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.650398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.650372 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.650398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.650390 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.650511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.650403 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.650511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.650434 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.652575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.652525 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-dshm" pod "f3b720de-005d-4af7-b356-2d2387bf4e57" (UID: "f3b720de-005d-4af7-b356-2d2387bf4e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.652575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.652536 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-dshm" pod "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" (UID: "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:32.652780 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.652617 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tls-certs" pod "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" (UID: "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:32.653080 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.653042 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b720de-005d-4af7-b356-2d2387bf4e57-tls-certs" pod "f3b720de-005d-4af7-b356-2d2387bf4e57" (UID: "f3b720de-005d-4af7-b356-2d2387bf4e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:32.653158 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.653141 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b720de-005d-4af7-b356-2d2387bf4e57-kube-api-access-22p7k" pod "f3b720de-005d-4af7-b356-2d2387bf4e57" (UID: "f3b720de-005d-4af7-b356-2d2387bf4e57"). InnerVolumeSpecName "kube-api-access-22p7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:32.653199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.653170 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-kube-api-access-f7jmk" pod "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" (UID: "c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de"). InnerVolumeSpecName "kube-api-access-f7jmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:32.751341 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.751307 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.751341 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.751335 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.751341 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.751344 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b720de-005d-4af7-b356-2d2387bf4e57-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.751584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.751352 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.751584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.751362 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22p7k\" (UniqueName: \"kubernetes.io/projected/f3b720de-005d-4af7-b356-2d2387bf4e57-kube-api-access-22p7k\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.751584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.751371 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7jmk\" (UniqueName: \"kubernetes.io/projected/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de-kube-api-access-f7jmk\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.751584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.751379 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b720de-005d-4af7-b356-2d2387bf4e57-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:32.817183 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.817148 5272 generic.go:356] "Generic (PLEG): container finished" podID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerID="bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9" exitCode=0 Nov 03 23:06:32.817637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.817195 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" Nov 03 23:06:32.817637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.817233 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" event={"ID":"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de","Type":"ContainerDied","Data":"bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9"} Nov 03 23:06:32.817637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.817288 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw" event={"ID":"c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de","Type":"ContainerDied","Data":"7e5d524aed5f0e7bf67ba8a882f223d036f5706593274af2593a0778e99fdc4e"} Nov 03 23:06:32.817637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.817311 5272 scope.go:122] "RemoveContainer" containerID="bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9" Nov 03 23:06:32.819038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.819015 5272 generic.go:356] "Generic (PLEG): container finished" podID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerID="7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989" exitCode=0 Nov 03 23:06:32.819158 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.819073 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" Nov 03 23:06:32.819158 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.819101 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" event={"ID":"f3b720de-005d-4af7-b356-2d2387bf4e57","Type":"ContainerDied","Data":"7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989"} Nov 03 23:06:32.819274 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.819157 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv" event={"ID":"f3b720de-005d-4af7-b356-2d2387bf4e57","Type":"ContainerDied","Data":"f97ee13b30cea1005655e136c4240f127a3aadaf55e7f81d7b1a0634251bc2d0"} Nov 03 23:06:32.821226 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.821201 5272 generic.go:356] "Generic (PLEG): container finished" podID="d0777fdd-20ab-471d-9a37-42a03e4d9ddd" containerID="8c9f87c4335f19bc6896ed38cda51a843f13b7133645107044a704d0abb93cf6" exitCode=0 Nov 03 23:06:32.821333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.821269 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" event={"ID":"d0777fdd-20ab-471d-9a37-42a03e4d9ddd","Type":"ContainerDied","Data":"8c9f87c4335f19bc6896ed38cda51a843f13b7133645107044a704d0abb93cf6"} Nov 03 23:06:32.828626 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.828605 5272 scope.go:122] "RemoveContainer" containerID="bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9" Nov 03 23:06:32.829004 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:06:32.828984 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9\": container with ID starting with bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9 not found: ID does not exist" containerID="bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9" Nov 03 23:06:32.829086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.829010 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9"} err="failed to get container status \"bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9\": rpc error: code = NotFound desc = could not find container \"bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9\": container with ID starting with bad0a43e81756bc7520505fd573644769a720c10c5213a44eb6c9e64a7ab33f9 not found: ID does not exist" Nov 03 23:06:32.829086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.829030 5272 scope.go:122] "RemoveContainer" containerID="7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989" Nov 03 23:06:32.838093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.838071 5272 scope.go:122] "RemoveContainer" containerID="7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989" Nov 03 23:06:32.838378 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:06:32.838359 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989\": container with ID starting with 7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989 not found: ID does not exist" containerID="7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989" Nov 03 23:06:32.838483 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.838386 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989"} err="failed to get container status \"7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989\": rpc error: code = NotFound desc = could not find container \"7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989\": container with ID starting with 7ce219c4a0c078ed45eb413c9943983e78e678a68409174deea6ec82dadf0989 not found: ID does not exist" Nov 03 23:06:32.971462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.971425 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv"] Nov 03 23:06:32.979049 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:32.979015 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/distinct-w6-5e72ad-kserve-58f549d47c-q2lsv"] Nov 03 23:06:33.187436 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:33.187335 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw"] Nov 03 23:06:33.192930 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:33.192894 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5e72ad/simulator-w6-5e72ad-kserve-98c947c5f-7gvvw"] Nov 03 23:06:33.348992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:33.348955 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:06:33.492934 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:33.492857 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" path="/var/lib/kubelet/pods/c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de/volumes" Nov 03 23:06:33.493361 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:33.493345 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" path="/var/lib/kubelet/pods/f3b720de-005d-4af7-b356-2d2387bf4e57/volumes" Nov 03 23:06:34.041374 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.041340 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:34.060861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.059990 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5"] Nov 03 23:06:34.063957 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.063921 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116d4246-xn4w5"] Nov 03 23:06:34.064138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.064067 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-openshift-ca\") pod \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " Nov 03 23:06:34.064138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.064129 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-kube-api-access-dxpxf\" (UniqueName: \"kubernetes.io/projected/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-kube-api-access-dxpxf\") pod \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " Nov 03 23:06:34.064392 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.064366 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-xks-ca\") pod \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\" (UID: \"d0777fdd-20ab-471d-9a37-42a03e4d9ddd\") " Nov 03 23:06:34.064514 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.064483 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-openshift-ca" pod "d0777fdd-20ab-471d-9a37-42a03e4d9ddd" (UID: "d0777fdd-20ab-471d-9a37-42a03e4d9ddd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:34.064702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.064685 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:34.066600 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.066569 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-xks-ca" pod "d0777fdd-20ab-471d-9a37-42a03e4d9ddd" (UID: "d0777fdd-20ab-471d-9a37-42a03e4d9ddd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:34.066844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.066795 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-kube-api-access-dxpxf" pod "d0777fdd-20ab-471d-9a37-42a03e4d9ddd" (UID: "d0777fdd-20ab-471d-9a37-42a03e4d9ddd"). InnerVolumeSpecName "kube-api-access-dxpxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:34.166255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.166175 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:34.166255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.166208 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxpxf\" (UniqueName: \"kubernetes.io/projected/d0777fdd-20ab-471d-9a37-42a03e4d9ddd-kube-api-access-dxpxf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:34.843998 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.843971 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" Nov 03 23:06:34.844189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.844009 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2" event={"ID":"d0777fdd-20ab-471d-9a37-42a03e4d9ddd","Type":"ContainerDied","Data":"54a089155d242955978ef9afd2df99631a7123eea9bfa8bafcb17b7be5521464"} Nov 03 23:06:34.844189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:34.844055 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a089155d242955978ef9afd2df99631a7123eea9bfa8bafcb17b7be5521464" Nov 03 23:06:35.492996 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:35.492960 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c47846-2bf7-4bb6-a407-d31ecb75f2df" path="/var/lib/kubelet/pods/b7c47846-2bf7-4bb6-a407-d31ecb75f2df/volumes" Nov 03 23:06:43.659706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:43.659668 5272 scope.go:122] "RemoveContainer" containerID="65dd545d32bbaf27c8783313a23dd267d6499f304e6b0f0a8dcfbd07a4509bae" Nov 03 23:06:43.672046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:43.672020 5272 scope.go:122] "RemoveContainer" containerID="81950b2487bf3dc6030733753036afc77d2da7db5e300ad3727847c30111a4fa" Nov 03 23:06:44.620992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:44.620943 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp"] Nov 03 23:06:44.733161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:44.733107 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn"] Nov 03 23:06:48.352932 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.352897 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv"] Nov 03 23:06:48.353384 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353371 5272 cpu_manager.go:414] "containerMap: removing container" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" Nov 03 23:06:48.353430 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353385 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" Nov 03 23:06:48.353469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353429 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d0777fdd-20ab-471d-9a37-42a03e4d9ddd" containerName="revoke-keys" Nov 03 23:06:48.353469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353438 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0777fdd-20ab-471d-9a37-42a03e4d9ddd" containerName="revoke-keys" Nov 03 23:06:48.353469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353463 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" Nov 03 23:06:48.353605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353473 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" Nov 03 23:06:48.353605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353587 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0daab0f-a22b-4a3a-b54e-23a6ccf9b1de" containerName="main" Nov 03 23:06:48.353605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353602 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b720de-005d-4af7-b356-2d2387bf4e57" containerName="main" Nov 03 23:06:48.353774 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.353613 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0777fdd-20ab-471d-9a37-42a03e4d9ddd" containerName="revoke-keys" Nov 03 23:06:48.358589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.358570 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.408757 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.408727 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:06:48.410602 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.410574 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv"] Nov 03 23:06:48.423395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.423359 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.423608 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.423482 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.423608 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.423587 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrr6\" (UniqueName: \"kubernetes.io/projected/bb1b5305-3e01-43cd-aad8-25756270a140-kube-api-access-5xrr6\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.453734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.453700 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:48.453894 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.453788 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:48.454984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.454965 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:48.457798 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.457777 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv"] Nov 03 23:06:48.461606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.461591 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.518730 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.518317 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv"] Nov 03 23:06:48.525042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.525014 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrr6\" (UniqueName: \"kubernetes.io/projected/bb1b5305-3e01-43cd-aad8-25756270a140-kube-api-access-5xrr6\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.525202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.525059 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.525202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.525102 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.525202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.525130 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.525323 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.525212 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.525323 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.525263 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzj8\" (UniqueName: \"kubernetes.io/projected/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-kube-api-access-kpzj8\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.526384 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.526354 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.527831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.527790 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.582230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.582199 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrr6\" (UniqueName: \"kubernetes.io/projected/bb1b5305-3e01-43cd-aad8-25756270a140-kube-api-access-5xrr6\") pod \"maas-api-revoke-sub-11c96d85-w6lrv\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.626623 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.626550 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzj8\" (UniqueName: \"kubernetes.io/projected/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-kube-api-access-kpzj8\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.626937 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.626906 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.627082 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.626993 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.627444 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.627424 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.629145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.629127 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.671790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.671765 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:48.706506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.706468 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzj8\" (UniqueName: \"kubernetes.io/projected/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-kube-api-access-kpzj8\") pod \"maas-api-revoke-sub-10ef1667-vt6nv\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:48.775789 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:48.775738 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:49.077045 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.077011 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv"] Nov 03 23:06:49.083096 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:06:49.083067 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1b5305_3e01_43cd_aad8_25756270a140.slice/crio-cfb34fa88a568e5f844a424705e8a9071bbb857324aa5413ee0d81e839e52740 WatchSource:0}: Error finding container cfb34fa88a568e5f844a424705e8a9071bbb857324aa5413ee0d81e839e52740: Status 404 returned error can't find the container with id cfb34fa88a568e5f844a424705e8a9071bbb857324aa5413ee0d81e839e52740 Nov 03 23:06:49.181835 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.181785 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv"] Nov 03 23:06:49.188201 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:06:49.188172 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba6c351_f8c4_4f9c_9f94_4b8fc736d7f3.slice/crio-f46316508dc2e62944191b9a6688c3606e78ad6995d17036baa8c71bd5b07cc2 WatchSource:0}: Error finding container f46316508dc2e62944191b9a6688c3606e78ad6995d17036baa8c71bd5b07cc2: Status 404 returned error can't find the container with id f46316508dc2e62944191b9a6688c3606e78ad6995d17036baa8c71bd5b07cc2 Nov 03 23:06:49.621643 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.621544 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" podUID="ff009dd7-805d-4475-9eea-b93e4ccc4737" containerName="payload-processing" containerID="cri-o://c6a0392046a3d12de9f5559a41d3a5ee74c0bcae1c256dc400c70ee1eaf7aede" gracePeriod=25 Nov 03 23:06:49.738414 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.737878 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" podUID="77ab87b2-6e1a-4c03-ac68-1f8733368037" containerName="payload-pre-processing" containerID="cri-o://feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728" gracePeriod=25 Nov 03 23:06:49.968168 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.968125 5272 generic.go:356] "Generic (PLEG): container finished" podID="ff009dd7-805d-4475-9eea-b93e4ccc4737" containerID="c6a0392046a3d12de9f5559a41d3a5ee74c0bcae1c256dc400c70ee1eaf7aede" exitCode=0 Nov 03 23:06:49.968420 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.968309 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" event={"ID":"ff009dd7-805d-4475-9eea-b93e4ccc4737","Type":"ContainerDied","Data":"c6a0392046a3d12de9f5559a41d3a5ee74c0bcae1c256dc400c70ee1eaf7aede"} Nov 03 23:06:49.970865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.970828 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" event={"ID":"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3","Type":"ContainerStarted","Data":"6418502d62faa490b0056469c8c0ff15e1e31888ef86c6da2599b0f96e3e7ad0"} Nov 03 23:06:49.971077 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.971058 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" event={"ID":"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3","Type":"ContainerStarted","Data":"f46316508dc2e62944191b9a6688c3606e78ad6995d17036baa8c71bd5b07cc2"} Nov 03 23:06:49.974427 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.974399 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" event={"ID":"bb1b5305-3e01-43cd-aad8-25756270a140","Type":"ContainerStarted","Data":"4c283fa41c607e18850b5fa82e49ae59e22005ce606c9039fbc9d4312229677c"} Nov 03 23:06:49.974617 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:49.974599 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" event={"ID":"bb1b5305-3e01-43cd-aad8-25756270a140","Type":"ContainerStarted","Data":"cfb34fa88a568e5f844a424705e8a9071bbb857324aa5413ee0d81e839e52740"} Nov 03 23:06:50.038834 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.038792 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 23:06:50.081816 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.081747 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" podStartSLOduration=2.081734253 podStartE2EDuration="2.081734253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:50.079550782 +0000 UTC m=+2590.701859089" watchObservedRunningTime="2026-11-03 23:06:50.081734253 +0000 UTC m=+2590.704042560" Nov 03 23:06:50.164744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.164646 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff009dd7-805d-4475-9eea-b93e4ccc4737-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff009dd7-805d-4475-9eea-b93e4ccc4737-plugins-config-volume\") pod \"ff009dd7-805d-4475-9eea-b93e4ccc4737\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " Nov 03 23:06:50.164744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.164739 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff009dd7-805d-4475-9eea-b93e4ccc4737-kube-api-access-rmqm2\" (UniqueName: \"kubernetes.io/projected/ff009dd7-805d-4475-9eea-b93e4ccc4737-kube-api-access-rmqm2\") pod \"ff009dd7-805d-4475-9eea-b93e4ccc4737\" (UID: \"ff009dd7-805d-4475-9eea-b93e4ccc4737\") " Nov 03 23:06:50.165079 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.165054 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff009dd7-805d-4475-9eea-b93e4ccc4737-plugins-config-volume" pod "ff009dd7-805d-4475-9eea-b93e4ccc4737" (UID: "ff009dd7-805d-4475-9eea-b93e4ccc4737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:50.165238 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.165215 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff009dd7-805d-4475-9eea-b93e4ccc4737-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:50.166951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.166927 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff009dd7-805d-4475-9eea-b93e4ccc4737-kube-api-access-rmqm2" pod "ff009dd7-805d-4475-9eea-b93e4ccc4737" (UID: "ff009dd7-805d-4475-9eea-b93e4ccc4737"). InnerVolumeSpecName "kube-api-access-rmqm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:50.266312 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.266270 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqm2\" (UniqueName: \"kubernetes.io/projected/ff009dd7-805d-4475-9eea-b93e4ccc4737-kube-api-access-rmqm2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:50.300005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.299382 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" podStartSLOduration=2.299361714 podStartE2EDuration="2.299361714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:50.295332133 +0000 UTC m=+2590.917640442" watchObservedRunningTime="2026-11-03 23:06:50.299361714 +0000 UTC m=+2590.921670023" Nov 03 23:06:50.612441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.612406 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2"] Nov 03 23:06:50.615500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.615471 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b4383-d49s2"] Nov 03 23:06:50.982537 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.982442 5272 generic.go:356] "Generic (PLEG): container finished" podID="fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" containerID="6418502d62faa490b0056469c8c0ff15e1e31888ef86c6da2599b0f96e3e7ad0" exitCode=0 Nov 03 23:06:50.982977 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.982529 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" event={"ID":"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3","Type":"ContainerDied","Data":"6418502d62faa490b0056469c8c0ff15e1e31888ef86c6da2599b0f96e3e7ad0"} Nov 03 23:06:50.984224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.984199 5272 generic.go:356] "Generic (PLEG): container finished" podID="bb1b5305-3e01-43cd-aad8-25756270a140" containerID="4c283fa41c607e18850b5fa82e49ae59e22005ce606c9039fbc9d4312229677c" exitCode=0 Nov 03 23:06:50.984364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.984289 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" event={"ID":"bb1b5305-3e01-43cd-aad8-25756270a140","Type":"ContainerDied","Data":"4c283fa41c607e18850b5fa82e49ae59e22005ce606c9039fbc9d4312229677c"} Nov 03 23:06:50.985522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.985503 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" event={"ID":"ff009dd7-805d-4475-9eea-b93e4ccc4737","Type":"ContainerDied","Data":"0716f24488a69b81106346bfa3d42a6d1c12b8196edbc908426762b900a6e1f5"} Nov 03 23:06:50.985602 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.985543 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp" Nov 03 23:06:50.985602 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:50.985552 5272 scope.go:122] "RemoveContainer" containerID="c6a0392046a3d12de9f5559a41d3a5ee74c0bcae1c256dc400c70ee1eaf7aede" Nov 03 23:06:51.306652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:51.306602 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp"] Nov 03 23:06:51.309382 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:51.309344 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9b4383-5b9cf487f8-qlxxp"] Nov 03 23:06:51.492219 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:51.492186 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0777fdd-20ab-471d-9a37-42a03e4d9ddd" path="/var/lib/kubelet/pods/d0777fdd-20ab-471d-9a37-42a03e4d9ddd/volumes" Nov 03 23:06:51.492596 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:51.492582 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff009dd7-805d-4475-9eea-b93e4ccc4737" path="/var/lib/kubelet/pods/ff009dd7-805d-4475-9eea-b93e4ccc4737/volumes" Nov 03 23:06:52.165070 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.165047 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:52.187179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.187149 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:52.287195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.287164 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-openshift-ca\") pod \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " Nov 03 23:06:52.287404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.287204 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-kube-api-access-kpzj8\" (UniqueName: \"kubernetes.io/projected/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-kube-api-access-kpzj8\") pod \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " Nov 03 23:06:52.287404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.287306 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1b5305-3e01-43cd-aad8-25756270a140-kube-api-access-5xrr6\" (UniqueName: \"kubernetes.io/projected/bb1b5305-3e01-43cd-aad8-25756270a140-kube-api-access-5xrr6\") pod \"bb1b5305-3e01-43cd-aad8-25756270a140\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " Nov 03 23:06:52.287404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.287378 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-xks-ca\") pod \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\" (UID: \"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3\") " Nov 03 23:06:52.287575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.287432 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-xks-ca\") pod \"bb1b5305-3e01-43cd-aad8-25756270a140\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " Nov 03 23:06:52.287575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.287551 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-openshift-ca\") pod \"bb1b5305-3e01-43cd-aad8-25756270a140\" (UID: \"bb1b5305-3e01-43cd-aad8-25756270a140\") " Nov 03 23:06:52.287702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.287571 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-openshift-ca" pod "fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" (UID: "fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:52.288283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.288252 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:52.288469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.288443 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-openshift-ca" pod "bb1b5305-3e01-43cd-aad8-25756270a140" (UID: "bb1b5305-3e01-43cd-aad8-25756270a140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:52.290956 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.290925 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-xks-ca" pod "fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" (UID: "fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:52.291061 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.290958 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-xks-ca" pod "bb1b5305-3e01-43cd-aad8-25756270a140" (UID: "bb1b5305-3e01-43cd-aad8-25756270a140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:52.291061 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.290964 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1b5305-3e01-43cd-aad8-25756270a140-kube-api-access-5xrr6" pod "bb1b5305-3e01-43cd-aad8-25756270a140" (UID: "bb1b5305-3e01-43cd-aad8-25756270a140"). InnerVolumeSpecName "kube-api-access-5xrr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:52.291061 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.290982 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-kube-api-access-kpzj8" pod "fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" (UID: "fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3"). InnerVolumeSpecName "kube-api-access-kpzj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:52.389303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.389261 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:52.389303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.389304 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzj8\" (UniqueName: \"kubernetes.io/projected/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-kube-api-access-kpzj8\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:52.389303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.389318 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xrr6\" (UniqueName: \"kubernetes.io/projected/bb1b5305-3e01-43cd-aad8-25756270a140-kube-api-access-5xrr6\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:52.389303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.389334 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:52.389665 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:52.389349 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1b5305-3e01-43cd-aad8-25756270a140-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:06:53.002035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:53.002000 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" Nov 03 23:06:53.002035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:53.002022 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv" event={"ID":"fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3","Type":"ContainerDied","Data":"f46316508dc2e62944191b9a6688c3606e78ad6995d17036baa8c71bd5b07cc2"} Nov 03 23:06:53.002292 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:53.002077 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46316508dc2e62944191b9a6688c3606e78ad6995d17036baa8c71bd5b07cc2" Nov 03 23:06:53.003522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:53.003481 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" event={"ID":"bb1b5305-3e01-43cd-aad8-25756270a140","Type":"ContainerDied","Data":"cfb34fa88a568e5f844a424705e8a9071bbb857324aa5413ee0d81e839e52740"} Nov 03 23:06:53.003522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:53.003509 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv" Nov 03 23:06:53.003664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:53.003532 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb34fa88a568e5f844a424705e8a9071bbb857324aa5413ee0d81e839e52740" Nov 03 23:06:54.503706 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:06:54.503660 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:06:57.532129 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:57.532090 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9"] Nov 03 23:06:57.532590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:57.532291 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" containerID="cri-o://8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e" gracePeriod=30 Nov 03 23:06:58.111644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.111602 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" podUID="77ab87b2-6e1a-4c03-ac68-1f8733368037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:58.333058 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333026 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs"] Nov 03 23:06:58.333550 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333535 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" containerName="revoke-keys" Nov 03 23:06:58.333595 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333552 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" containerName="revoke-keys" Nov 03 23:06:58.333595 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333592 5272 cpu_manager.go:414] "containerMap: removing container" podUID="bb1b5305-3e01-43cd-aad8-25756270a140" containerName="revoke-keys" Nov 03 23:06:58.333667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333598 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1b5305-3e01-43cd-aad8-25756270a140" containerName="revoke-keys" Nov 03 23:06:58.333667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333617 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ff009dd7-805d-4475-9eea-b93e4ccc4737" containerName="payload-processing" Nov 03 23:06:58.333667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333621 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff009dd7-805d-4475-9eea-b93e4ccc4737" containerName="payload-processing" Nov 03 23:06:58.333760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333701 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1b5305-3e01-43cd-aad8-25756270a140" containerName="revoke-keys" Nov 03 23:06:58.333760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333713 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff009dd7-805d-4475-9eea-b93e4ccc4737" containerName="payload-processing" Nov 03 23:06:58.333760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.333726 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" containerName="revoke-keys" Nov 03 23:06:58.337446 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.337420 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.388194 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.388078 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:58.391283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.391260 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs"] Nov 03 23:06:58.434455 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.434421 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:58.434455 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.434434 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:58.434644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.434530 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:06:58.459676 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.459646 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.459850 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.459695 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjr5\" (UniqueName: \"kubernetes.io/projected/6663893d-3bc4-4580-a350-bb5705a94abd-kube-api-access-lmjr5\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.459898 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.459864 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.561190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.561142 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.561835 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.561338 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.561835 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.561385 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjr5\" (UniqueName: \"kubernetes.io/projected/6663893d-3bc4-4580-a350-bb5705a94abd-kube-api-access-lmjr5\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.562031 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.562012 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.563624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.563599 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.618759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.618725 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjr5\" (UniqueName: \"kubernetes.io/projected/6663893d-3bc4-4580-a350-bb5705a94abd-kube-api-access-lmjr5\") pod \"maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:58.651083 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:58.651015 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:06:59.023982 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:59.023900 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs"] Nov 03 23:06:59.030066 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:06:59.030035 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6663893d_3bc4_4580_a350_bb5705a94abd.slice/crio-701b047f57c3b70cf54be0567e5f5f42209be89dffd59dfb523432c52a0cc8e7 WatchSource:0}: Error finding container 701b047f57c3b70cf54be0567e5f5f42209be89dffd59dfb523432c52a0cc8e7: Status 404 returned error can't find the container with id 701b047f57c3b70cf54be0567e5f5f42209be89dffd59dfb523432c52a0cc8e7 Nov 03 23:06:59.047541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:06:59.047516 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" event={"ID":"6663893d-3bc4-4580-a350-bb5705a94abd","Type":"ContainerStarted","Data":"701b047f57c3b70cf54be0567e5f5f42209be89dffd59dfb523432c52a0cc8e7"} Nov 03 23:07:00.055663 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:00.055633 5272 generic.go:356] "Generic (PLEG): container finished" podID="6663893d-3bc4-4580-a350-bb5705a94abd" containerID="416cf343b78f49f40475e5a380d5633f71c43007191ca5bd228295c79c955acf" exitCode=0 Nov 03 23:07:00.056045 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:00.055716 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" event={"ID":"6663893d-3bc4-4580-a350-bb5705a94abd","Type":"ContainerDied","Data":"416cf343b78f49f40475e5a380d5633f71c43007191ca5bd228295c79c955acf"} Nov 03 23:07:01.201945 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.201920 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:07:01.292956 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.292925 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-openshift-ca\") pod \"6663893d-3bc4-4580-a350-bb5705a94abd\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " Nov 03 23:07:01.292956 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.292959 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6663893d-3bc4-4580-a350-bb5705a94abd-kube-api-access-lmjr5\" (UniqueName: \"kubernetes.io/projected/6663893d-3bc4-4580-a350-bb5705a94abd-kube-api-access-lmjr5\") pod \"6663893d-3bc4-4580-a350-bb5705a94abd\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " Nov 03 23:07:01.293232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.292983 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-xks-ca\") pod \"6663893d-3bc4-4580-a350-bb5705a94abd\" (UID: \"6663893d-3bc4-4580-a350-bb5705a94abd\") " Nov 03 23:07:01.293320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.293288 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-openshift-ca" pod "6663893d-3bc4-4580-a350-bb5705a94abd" (UID: "6663893d-3bc4-4580-a350-bb5705a94abd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:01.295200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.295170 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6663893d-3bc4-4580-a350-bb5705a94abd-kube-api-access-lmjr5" pod "6663893d-3bc4-4580-a350-bb5705a94abd" (UID: "6663893d-3bc4-4580-a350-bb5705a94abd"). InnerVolumeSpecName "kube-api-access-lmjr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:01.295200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.295172 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-xks-ca" pod "6663893d-3bc4-4580-a350-bb5705a94abd" (UID: "6663893d-3bc4-4580-a350-bb5705a94abd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:01.394363 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.394284 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:01.394363 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.394313 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmjr5\" (UniqueName: \"kubernetes.io/projected/6663893d-3bc4-4580-a350-bb5705a94abd-kube-api-access-lmjr5\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:01.394363 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:01.394326 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6663893d-3bc4-4580-a350-bb5705a94abd-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.069541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.069509 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" event={"ID":"6663893d-3bc4-4580-a350-bb5705a94abd","Type":"ContainerDied","Data":"701b047f57c3b70cf54be0567e5f5f42209be89dffd59dfb523432c52a0cc8e7"} Nov 03 23:07:02.069541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.069538 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs" Nov 03 23:07:02.069754 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.069558 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="701b047f57c3b70cf54be0567e5f5f42209be89dffd59dfb523432c52a0cc8e7" Nov 03 23:07:02.793655 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.793629 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 23:07:02.912545 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912457 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-data\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.912545 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912527 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-kube-api-access-tnllg\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-kube-api-access-tnllg\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.912769 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912602 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-socket\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.912769 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912634 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-envoy\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.912769 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912741 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-credential-socket\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.912973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912790 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-certs\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.912973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912830 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-data" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:02.912973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912942 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c27d9423-3e14-44de-a112-3edc286ab845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27d9423-3e14-44de-a112-3edc286ab845-istio-podinfo\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.913161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912955 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-socket" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:02.913161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.912985 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27d9423-3e14-44de-a112-3edc286ab845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27d9423-3e14-44de-a112-3edc286ab845-istiod-ca-cert\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.913161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913048 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-istio-token\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-istio-token\") pod \"c27d9423-3e14-44de-a112-3edc286ab845\" (UID: \"c27d9423-3e14-44de-a112-3edc286ab845\") " Nov 03 23:07:02.913161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913039 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-credential-socket" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:02.913161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913106 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-certs" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:02.913439 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913306 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27d9423-3e14-44de-a112-3edc286ab845-istiod-ca-cert" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:02.913561 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913541 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.913641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913564 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.913641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913578 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27d9423-3e14-44de-a112-3edc286ab845-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.913641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913594 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.913641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.913607 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.915098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.915047 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-envoy" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:02.915229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.915147 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-istio-token" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:02.915339 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.915321 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-kube-api-access-tnllg" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "kube-api-access-tnllg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:02.915427 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:02.915412 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c27d9423-3e14-44de-a112-3edc286ab845-istio-podinfo" pod "c27d9423-3e14-44de-a112-3edc286ab845" (UID: "c27d9423-3e14-44de-a112-3edc286ab845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:03.015189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.015154 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27d9423-3e14-44de-a112-3edc286ab845-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.015189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.015182 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.015189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.015193 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnllg\" (UniqueName: \"kubernetes.io/projected/c27d9423-3e14-44de-a112-3edc286ab845-kube-api-access-tnllg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.015434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.015203 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27d9423-3e14-44de-a112-3edc286ab845-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.077582 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.077545 5272 generic.go:356] "Generic (PLEG): container finished" podID="c27d9423-3e14-44de-a112-3edc286ab845" containerID="8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e" exitCode=0 Nov 03 23:07:03.077775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.077594 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" Nov 03 23:07:03.077775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.077625 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" event={"ID":"c27d9423-3e14-44de-a112-3edc286ab845","Type":"ContainerDied","Data":"8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e"} Nov 03 23:07:03.077775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.077734 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9" event={"ID":"c27d9423-3e14-44de-a112-3edc286ab845","Type":"ContainerDied","Data":"b05853dc65daa1290b3adb5da707f011dc07e9baa5166dd4e39caa767c10f367"} Nov 03 23:07:03.077775 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.077766 5272 scope.go:122] "RemoveContainer" containerID="8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e" Nov 03 23:07:03.088967 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.088953 5272 scope.go:122] "RemoveContainer" containerID="409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1" Nov 03 23:07:03.097541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.097516 5272 scope.go:122] "RemoveContainer" containerID="8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e" Nov 03 23:07:03.097769 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:03.097753 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e\": container with ID starting with 8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e not found: ID does not exist" containerID="8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e" Nov 03 23:07:03.097861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.097777 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e"} err="failed to get container status \"8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e\": rpc error: code = NotFound desc = could not find container \"8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e\": container with ID starting with 8265edf098a071d5b6cd3ced8b41e8150e5818aa519a187f7a0f6392a13d938e not found: ID does not exist" Nov 03 23:07:03.097861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.097816 5272 scope.go:122] "RemoveContainer" containerID="409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1" Nov 03 23:07:03.098037 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:03.098023 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1\": container with ID starting with 409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1 not found: ID does not exist" containerID="409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1" Nov 03 23:07:03.098084 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.098041 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1"} err="failed to get container status \"409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1\": rpc error: code = NotFound desc = could not find container \"409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1\": container with ID starting with 409f061cf3afe716553e922376c1d8b14ebcbf01c4994e48df86ac48ebd99cd1 not found: ID does not exist" Nov 03 23:07:03.232699 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.232617 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9"] Nov 03 23:07:03.236692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.236666 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b4383-openshift-default-64f46d78c7-lblz9"] Nov 03 23:07:03.493093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:03.493015 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27d9423-3e14-44de-a112-3edc286ab845" path="/var/lib/kubelet/pods/c27d9423-3e14-44de-a112-3edc286ab845/volumes" Nov 03 23:07:08.111251 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:08.111215 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" podUID="77ab87b2-6e1a-4c03-ac68-1f8733368037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:07:08.523575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:08.523490 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:07:09.877160 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:09.877125 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7"] Nov 03 23:07:09.905492 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:09.905463 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 23:07:09.976746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:09.976706 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf"] Nov 03 23:07:09.993953 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:09.993924 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ab87b2-6e1a-4c03-ac68-1f8733368037-kube-api-access-jjmr7\" (UniqueName: \"kubernetes.io/projected/77ab87b2-6e1a-4c03-ac68-1f8733368037-kube-api-access-jjmr7\") pod \"77ab87b2-6e1a-4c03-ac68-1f8733368037\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " Nov 03 23:07:09.994126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:09.994021 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ab87b2-6e1a-4c03-ac68-1f8733368037-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ab87b2-6e1a-4c03-ac68-1f8733368037-plugins-config-volume\") pod \"77ab87b2-6e1a-4c03-ac68-1f8733368037\" (UID: \"77ab87b2-6e1a-4c03-ac68-1f8733368037\") " Nov 03 23:07:09.994457 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:09.994435 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ab87b2-6e1a-4c03-ac68-1f8733368037-plugins-config-volume" pod "77ab87b2-6e1a-4c03-ac68-1f8733368037" (UID: "77ab87b2-6e1a-4c03-ac68-1f8733368037"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:09.996117 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:09.996091 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ab87b2-6e1a-4c03-ac68-1f8733368037-kube-api-access-jjmr7" pod "77ab87b2-6e1a-4c03-ac68-1f8733368037" (UID: "77ab87b2-6e1a-4c03-ac68-1f8733368037"). InnerVolumeSpecName "kube-api-access-jjmr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:10.049751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.049706 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb"] Nov 03 23:07:10.054131 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.054091 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13be7e9e-qcdjb"] Nov 03 23:07:10.056725 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.056694 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76"] Nov 03 23:07:10.062293 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.062270 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4377ebff-l2j76"] Nov 03 23:07:10.095472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.095433 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjmr7\" (UniqueName: \"kubernetes.io/projected/77ab87b2-6e1a-4c03-ac68-1f8733368037-kube-api-access-jjmr7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:10.095472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.095474 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77ab87b2-6e1a-4c03-ac68-1f8733368037-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:10.125059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.125019 5272 generic.go:356] "Generic (PLEG): container finished" podID="77ab87b2-6e1a-4c03-ac68-1f8733368037" containerID="feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728" exitCode=0 Nov 03 23:07:10.125213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.125069 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" Nov 03 23:07:10.125213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.125104 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" event={"ID":"77ab87b2-6e1a-4c03-ac68-1f8733368037","Type":"ContainerDied","Data":"feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728"} Nov 03 23:07:10.125213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.125164 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn" event={"ID":"77ab87b2-6e1a-4c03-ac68-1f8733368037","Type":"ContainerDied","Data":"a42ff71deb17054e39c72c139411332ee22059b6f235511006691ec57feca35f"} Nov 03 23:07:10.125213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.125186 5272 scope.go:122] "RemoveContainer" containerID="feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728" Nov 03 23:07:10.135349 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.135330 5272 scope.go:122] "RemoveContainer" containerID="feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728" Nov 03 23:07:10.135634 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:10.135612 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728\": container with ID starting with feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728 not found: ID does not exist" containerID="feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728" Nov 03 23:07:10.135683 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.135644 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728"} err="failed to get container status \"feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728\": rpc error: code = NotFound desc = could not find container \"feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728\": container with ID starting with feb2eb81cc30b40994dbb1addcbc0fffb88894b9f05073ed31cb9740cee54728 not found: ID does not exist" Nov 03 23:07:10.340070 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.336969 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn"] Nov 03 23:07:10.340276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:10.340201 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9b4383-7c7d96654c-wj9bn"] Nov 03 23:07:11.048714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.048671 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq"] Nov 03 23:07:11.054070 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.054036 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls"] Nov 03 23:07:11.057523 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.057492 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f65e2e-ftxhq"] Nov 03 23:07:11.059629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.059598 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b7d52d-52sls"] Nov 03 23:07:11.493135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.493061 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084a5e7e-1c6a-4342-94e1-0c7517486c1b" path="/var/lib/kubelet/pods/084a5e7e-1c6a-4342-94e1-0c7517486c1b/volumes" Nov 03 23:07:11.493462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.493448 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6db47f-e8c1-4ed8-8a31-bbc28e01e958" path="/var/lib/kubelet/pods/2f6db47f-e8c1-4ed8-8a31-bbc28e01e958/volumes" Nov 03 23:07:11.493823 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.493796 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ab87b2-6e1a-4c03-ac68-1f8733368037" path="/var/lib/kubelet/pods/77ab87b2-6e1a-4c03-ac68-1f8733368037/volumes" Nov 03 23:07:11.494484 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.494469 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8997735e-2b8b-4d54-87b4-d881a374c7e1" path="/var/lib/kubelet/pods/8997735e-2b8b-4d54-87b4-d881a374c7e1/volumes" Nov 03 23:07:11.494874 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:11.494862 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d835ff5a-6984-4024-8234-f5e7e20bec9f" path="/var/lib/kubelet/pods/d835ff5a-6984-4024-8234-f5e7e20bec9f/volumes" Nov 03 23:07:14.877462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:14.877423 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" podUID="c6f5ee5b-5fae-4767-b899-7edade51d604" containerName="payload-processing" containerID="cri-o://fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a" gracePeriod=25 Nov 03 23:07:14.977471 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:14.977422 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" podUID="8cc28bb0-1698-4eaa-9238-0c9313875664" containerName="payload-pre-processing" containerID="cri-o://47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3" gracePeriod=25 Nov 03 23:07:15.124358 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.124326 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 23:07:15.162705 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.162629 5272 generic.go:356] "Generic (PLEG): container finished" podID="c6f5ee5b-5fae-4767-b899-7edade51d604" containerID="fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a" exitCode=0 Nov 03 23:07:15.162705 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.162676 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" Nov 03 23:07:15.162911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.162709 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" event={"ID":"c6f5ee5b-5fae-4767-b899-7edade51d604","Type":"ContainerDied","Data":"fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a"} Nov 03 23:07:15.162911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.162769 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7" event={"ID":"c6f5ee5b-5fae-4767-b899-7edade51d604","Type":"ContainerDied","Data":"ef24d4bc339c975f0930e8165be6d952032ac6e5a569b6b36a050686de44c370"} Nov 03 23:07:15.162911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.162791 5272 scope.go:122] "RemoveContainer" containerID="fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a" Nov 03 23:07:15.172526 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.172499 5272 scope.go:122] "RemoveContainer" containerID="fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a" Nov 03 23:07:15.172851 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:15.172788 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a\": container with ID starting with fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a not found: ID does not exist" containerID="fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a" Nov 03 23:07:15.172926 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.172861 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a"} err="failed to get container status \"fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a\": rpc error: code = NotFound desc = could not find container \"fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a\": container with ID starting with fb928053f8ec125b272a83b4c913166e5353551733005ce8437bbdfe57a8ec9a not found: ID does not exist" Nov 03 23:07:15.261737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.261701 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f5ee5b-5fae-4767-b899-7edade51d604-kube-api-access-pdgwh\" (UniqueName: \"kubernetes.io/projected/c6f5ee5b-5fae-4767-b899-7edade51d604-kube-api-access-pdgwh\") pod \"c6f5ee5b-5fae-4767-b899-7edade51d604\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " Nov 03 23:07:15.261939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.261911 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f5ee5b-5fae-4767-b899-7edade51d604-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5ee5b-5fae-4767-b899-7edade51d604-plugins-config-volume\") pod \"c6f5ee5b-5fae-4767-b899-7edade51d604\" (UID: \"c6f5ee5b-5fae-4767-b899-7edade51d604\") " Nov 03 23:07:15.262351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.262326 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f5ee5b-5fae-4767-b899-7edade51d604-plugins-config-volume" pod "c6f5ee5b-5fae-4767-b899-7edade51d604" (UID: "c6f5ee5b-5fae-4767-b899-7edade51d604"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:15.263897 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.263873 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f5ee5b-5fae-4767-b899-7edade51d604-kube-api-access-pdgwh" pod "c6f5ee5b-5fae-4767-b899-7edade51d604" (UID: "c6f5ee5b-5fae-4767-b899-7edade51d604"). InnerVolumeSpecName "kube-api-access-pdgwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:15.285150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.285124 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6"] Nov 03 23:07:15.285399 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.285376 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" containerID="cri-o://d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068" gracePeriod=30 Nov 03 23:07:15.363086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.363046 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5ee5b-5fae-4767-b899-7edade51d604-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:15.363086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.363079 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdgwh\" (UniqueName: \"kubernetes.io/projected/c6f5ee5b-5fae-4767-b899-7edade51d604-kube-api-access-pdgwh\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:15.577188 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.577149 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7"] Nov 03 23:07:15.580796 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:15.580764 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2cea18-6fbbf7d74b-wknm7"] Nov 03 23:07:17.492478 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:17.492443 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f5ee5b-5fae-4767-b899-7edade51d604" path="/var/lib/kubelet/pods/c6f5ee5b-5fae-4767-b899-7edade51d604/volumes" Nov 03 23:07:20.102696 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.102457 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" podUID="8cc28bb0-1698-4eaa-9238-0c9313875664" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:07:20.589047 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.589011 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 23:07:20.729378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729293 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-credential-socket\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729363 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-podinfo\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729387 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-token\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-token\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729436 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-kube-api-access-vbt4z\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-kube-api-access-vbt4z\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729493 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37834fae-53d5-4f2d-b1ef-c59ea9593019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37834fae-53d5-4f2d-b1ef-c59ea9593019-istiod-ca-cert\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729594 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-credential-socket" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.729637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729599 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-socket\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729967 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729670 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-envoy\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729967 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729771 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-certs\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.729967 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729830 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-socket" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.729967 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729841 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-data\") pod \"37834fae-53d5-4f2d-b1ef-c59ea9593019\" (UID: \"37834fae-53d5-4f2d-b1ef-c59ea9593019\") " Nov 03 23:07:20.730170 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.730057 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-certs" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.730170 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.729985 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37834fae-53d5-4f2d-b1ef-c59ea9593019-istiod-ca-cert" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:20.730285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.730233 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-data" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.730447 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.730422 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37834fae-53d5-4f2d-b1ef-c59ea9593019-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.730684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.730497 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.730684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.730536 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.730684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.730553 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.730684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.730566 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.731919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.731888 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-token" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:20.731919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.731907 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-podinfo" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:20.732123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.732105 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-kube-api-access-vbt4z" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "kube-api-access-vbt4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:20.732192 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.732176 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-envoy" pod "37834fae-53d5-4f2d-b1ef-c59ea9593019" (UID: "37834fae-53d5-4f2d-b1ef-c59ea9593019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.831990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.831954 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.831990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.831990 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.832201 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.832005 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbt4z\" (UniqueName: \"kubernetes.io/projected/37834fae-53d5-4f2d-b1ef-c59ea9593019-kube-api-access-vbt4z\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.832201 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:20.832021 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37834fae-53d5-4f2d-b1ef-c59ea9593019-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:21.206375 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.206340 5272 generic.go:356] "Generic (PLEG): container finished" podID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerID="d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068" exitCode=0 Nov 03 23:07:21.206831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.206390 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" Nov 03 23:07:21.206831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.206436 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" event={"ID":"37834fae-53d5-4f2d-b1ef-c59ea9593019","Type":"ContainerDied","Data":"d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068"} Nov 03 23:07:21.206831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.206538 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6" event={"ID":"37834fae-53d5-4f2d-b1ef-c59ea9593019","Type":"ContainerDied","Data":"bfed534402c9cdeb2102c43bb5b9c9bc38785d00a8f6fd937ea71e5d580e08d9"} Nov 03 23:07:21.206831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.206570 5272 scope.go:122] "RemoveContainer" containerID="d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068" Nov 03 23:07:21.217267 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.217250 5272 scope.go:122] "RemoveContainer" containerID="d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068" Nov 03 23:07:21.217530 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:21.217511 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068\": container with ID starting with d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068 not found: ID does not exist" containerID="d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068" Nov 03 23:07:21.217578 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.217538 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068"} err="failed to get container status \"d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068\": rpc error: code = NotFound desc = could not find container \"d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068\": container with ID starting with d4289bb5353109d24c8025641c15b338fea5ed2dbcd519ddc150877dd1c79068 not found: ID does not exist" Nov 03 23:07:21.366041 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.366008 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6"] Nov 03 23:07:21.412620 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.412586 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-68fbd1-openshift-default-5f969f4f67-8t4p6"] Nov 03 23:07:21.491721 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.491642 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" path="/var/lib/kubelet/pods/37834fae-53d5-4f2d-b1ef-c59ea9593019/volumes" Nov 03 23:07:21.705821 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.703733 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn"] Nov 03 23:07:21.712526 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:21.712481 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5e72ad-fwpjn"] Nov 03 23:07:23.089475 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:23.089442 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs"] Nov 03 23:07:23.091246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:23.091224 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2cea18-tdjxs"] Nov 03 23:07:23.492033 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:23.491965 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6663893d-3bc4-4580-a350-bb5705a94abd" path="/var/lib/kubelet/pods/6663893d-3bc4-4580-a350-bb5705a94abd/volumes" Nov 03 23:07:23.492383 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:23.492369 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb570a4f-29de-4fab-adde-bdea617245c7" path="/var/lib/kubelet/pods/cb570a4f-29de-4fab-adde-bdea617245c7/volumes" Nov 03 23:07:24.499229 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:24.499194 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:07:30.092919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.092879 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" podUID="8cc28bb0-1698-4eaa-9238-0c9313875664" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:07:30.885324 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885289 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp"] Nov 03 23:07:30.885761 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885748 5272 cpu_manager.go:414] "containerMap: removing container" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" Nov 03 23:07:30.885818 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885762 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" Nov 03 23:07:30.885818 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885775 5272 cpu_manager.go:414] "containerMap: removing container" podUID="6663893d-3bc4-4580-a350-bb5705a94abd" containerName="revoke-keys" Nov 03 23:07:30.885818 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885780 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6663893d-3bc4-4580-a350-bb5705a94abd" containerName="revoke-keys" Nov 03 23:07:30.885818 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885794 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" Nov 03 23:07:30.885818 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885812 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885823 5272 cpu_manager.go:414] "containerMap: removing container" podUID="77ab87b2-6e1a-4c03-ac68-1f8733368037" containerName="payload-pre-processing" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885828 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ab87b2-6e1a-4c03-ac68-1f8733368037" containerName="payload-pre-processing" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885846 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885852 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885868 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c6f5ee5b-5fae-4767-b899-7edade51d604" containerName="payload-processing" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885873 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f5ee5b-5fae-4767-b899-7edade51d604" containerName="payload-processing" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885929 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885938 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f5ee5b-5fae-4767-b899-7edade51d604" containerName="payload-processing" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885945 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="37834fae-53d5-4f2d-b1ef-c59ea9593019" containerName="istio-proxy" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885951 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="6663893d-3bc4-4580-a350-bb5705a94abd" containerName="revoke-keys" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885958 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27d9423-3e14-44de-a112-3edc286ab845" containerName="istio-proxy" Nov 03 23:07:30.885993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.885964 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ab87b2-6e1a-4c03-ac68-1f8733368037" containerName="payload-pre-processing" Nov 03 23:07:30.890982 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.890962 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:30.943249 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.943224 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp"] Nov 03 23:07:30.959193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.959148 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:30.959329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.959198 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ph9\" (UniqueName: \"kubernetes.io/projected/54c590d3-3731-46ad-8c50-18d4a132f5c9-kube-api-access-d8ph9\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:30.959329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.959314 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:30.987032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.987003 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:30.987032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.987020 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:07:30.987184 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.987013 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:07:30.987184 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:30.987026 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:07:31.060433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.060400 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:31.060433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.060434 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ph9\" (UniqueName: \"kubernetes.io/projected/54c590d3-3731-46ad-8c50-18d4a132f5c9-kube-api-access-d8ph9\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:31.060638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.060540 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:31.061181 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.061159 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:31.062828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.062788 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:31.117659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.117637 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ph9\" (UniqueName: \"kubernetes.io/projected/54c590d3-3731-46ad-8c50-18d4a132f5c9-kube-api-access-d8ph9\") pod \"maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:31.204471 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.204405 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:31.584823 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:31.584772 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp"] Nov 03 23:07:31.591476 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:07:31.591448 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c590d3_3731_46ad_8c50_18d4a132f5c9.slice/crio-6c7fc8f462c6a4f4dd6c26ba39353e56f2b02056dc71965847b6ab4e58114851 WatchSource:0}: Error finding container 6c7fc8f462c6a4f4dd6c26ba39353e56f2b02056dc71965847b6ab4e58114851: Status 404 returned error can't find the container with id 6c7fc8f462c6a4f4dd6c26ba39353e56f2b02056dc71965847b6ab4e58114851 Nov 03 23:07:32.292090 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:32.292058 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" event={"ID":"54c590d3-3731-46ad-8c50-18d4a132f5c9","Type":"ContainerStarted","Data":"544a16405f75a685a95c199331984c9f152a232821be55e16cff8d8c83c05cd3"} Nov 03 23:07:32.292468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:32.292106 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" event={"ID":"54c590d3-3731-46ad-8c50-18d4a132f5c9","Type":"ContainerStarted","Data":"6c7fc8f462c6a4f4dd6c26ba39353e56f2b02056dc71965847b6ab4e58114851"} Nov 03 23:07:32.401200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:32.401146 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" podStartSLOduration=2.401133853 podStartE2EDuration="2.401133853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:32.399726263 +0000 UTC m=+2633.022034572" watchObservedRunningTime="2026-11-03 23:07:32.401133853 +0000 UTC m=+2633.023442151" Nov 03 23:07:33.300023 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:33.299991 5272 generic.go:356] "Generic (PLEG): container finished" podID="54c590d3-3731-46ad-8c50-18d4a132f5c9" containerID="544a16405f75a685a95c199331984c9f152a232821be55e16cff8d8c83c05cd3" exitCode=0 Nov 03 23:07:33.300484 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:33.300079 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" event={"ID":"54c590d3-3731-46ad-8c50-18d4a132f5c9","Type":"ContainerDied","Data":"544a16405f75a685a95c199331984c9f152a232821be55e16cff8d8c83c05cd3"} Nov 03 23:07:34.438919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.438896 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:34.499164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.499137 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-xks-ca\") pod \"54c590d3-3731-46ad-8c50-18d4a132f5c9\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " Nov 03 23:07:34.499287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.499270 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-openshift-ca\") pod \"54c590d3-3731-46ad-8c50-18d4a132f5c9\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " Nov 03 23:07:34.499391 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.499374 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c590d3-3731-46ad-8c50-18d4a132f5c9-kube-api-access-d8ph9\" (UniqueName: \"kubernetes.io/projected/54c590d3-3731-46ad-8c50-18d4a132f5c9-kube-api-access-d8ph9\") pod \"54c590d3-3731-46ad-8c50-18d4a132f5c9\" (UID: \"54c590d3-3731-46ad-8c50-18d4a132f5c9\") " Nov 03 23:07:34.499635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.499610 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-openshift-ca" pod "54c590d3-3731-46ad-8c50-18d4a132f5c9" (UID: "54c590d3-3731-46ad-8c50-18d4a132f5c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:34.499784 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.499760 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:34.501220 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.501191 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-xks-ca" pod "54c590d3-3731-46ad-8c50-18d4a132f5c9" (UID: "54c590d3-3731-46ad-8c50-18d4a132f5c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:34.501365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.501340 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c590d3-3731-46ad-8c50-18d4a132f5c9-kube-api-access-d8ph9" pod "54c590d3-3731-46ad-8c50-18d4a132f5c9" (UID: "54c590d3-3731-46ad-8c50-18d4a132f5c9"). InnerVolumeSpecName "kube-api-access-d8ph9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:34.600997 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.600913 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c590d3-3731-46ad-8c50-18d4a132f5c9-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:34.600997 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:34.600954 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ph9\" (UniqueName: \"kubernetes.io/projected/54c590d3-3731-46ad-8c50-18d4a132f5c9-kube-api-access-d8ph9\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.218620 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.218593 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 23:07:35.308441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.308407 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc28bb0-1698-4eaa-9238-0c9313875664-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc28bb0-1698-4eaa-9238-0c9313875664-plugins-config-volume\") pod \"8cc28bb0-1698-4eaa-9238-0c9313875664\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " Nov 03 23:07:35.308628 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.308545 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc28bb0-1698-4eaa-9238-0c9313875664-kube-api-access-gpdwb\" (UniqueName: \"kubernetes.io/projected/8cc28bb0-1698-4eaa-9238-0c9313875664-kube-api-access-gpdwb\") pod \"8cc28bb0-1698-4eaa-9238-0c9313875664\" (UID: \"8cc28bb0-1698-4eaa-9238-0c9313875664\") " Nov 03 23:07:35.308850 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.308795 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc28bb0-1698-4eaa-9238-0c9313875664-plugins-config-volume" pod "8cc28bb0-1698-4eaa-9238-0c9313875664" (UID: "8cc28bb0-1698-4eaa-9238-0c9313875664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:35.310574 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.310554 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc28bb0-1698-4eaa-9238-0c9313875664-kube-api-access-gpdwb" pod "8cc28bb0-1698-4eaa-9238-0c9313875664" (UID: "8cc28bb0-1698-4eaa-9238-0c9313875664"). InnerVolumeSpecName "kube-api-access-gpdwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:35.313558 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.313540 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" Nov 03 23:07:35.313655 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.313540 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp" event={"ID":"54c590d3-3731-46ad-8c50-18d4a132f5c9","Type":"ContainerDied","Data":"6c7fc8f462c6a4f4dd6c26ba39353e56f2b02056dc71965847b6ab4e58114851"} Nov 03 23:07:35.313721 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.313706 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7fc8f462c6a4f4dd6c26ba39353e56f2b02056dc71965847b6ab4e58114851" Nov 03 23:07:35.315040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.315019 5272 generic.go:356] "Generic (PLEG): container finished" podID="8cc28bb0-1698-4eaa-9238-0c9313875664" containerID="47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3" exitCode=0 Nov 03 23:07:35.315124 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.315048 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" Nov 03 23:07:35.315124 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.315072 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" event={"ID":"8cc28bb0-1698-4eaa-9238-0c9313875664","Type":"ContainerDied","Data":"47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3"} Nov 03 23:07:35.315124 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.315110 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf" event={"ID":"8cc28bb0-1698-4eaa-9238-0c9313875664","Type":"ContainerDied","Data":"6309a1d0d348f26e8ef2f9815bdddfadcb101647e9406001d04de08e22c3b327"} Nov 03 23:07:35.315261 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.315130 5272 scope.go:122] "RemoveContainer" containerID="47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3" Nov 03 23:07:35.325072 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.325011 5272 scope.go:122] "RemoveContainer" containerID="47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3" Nov 03 23:07:35.325344 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:35.325320 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3\": container with ID starting with 47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3 not found: ID does not exist" containerID="47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3" Nov 03 23:07:35.325433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.325349 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3"} err="failed to get container status \"47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3\": rpc error: code = NotFound desc = could not find container \"47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3\": container with ID starting with 47bf16603d61cfcc6bc79e5e53e1fa16ef68e72ddb530621fe2c9bfb9bfc98f3 not found: ID does not exist" Nov 03 23:07:35.377488 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.377461 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh"] Nov 03 23:07:35.409947 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.409922 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cc28bb0-1698-4eaa-9238-0c9313875664-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.409947 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.409946 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdwb\" (UniqueName: \"kubernetes.io/projected/8cc28bb0-1698-4eaa-9238-0c9313875664-kube-api-access-gpdwb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:35.493556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.493520 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf"] Nov 03 23:07:35.493556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:35.493562 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2cea18-5bcdcd4dff-bczwf"] Nov 03 23:07:37.492429 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:37.492394 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc28bb0-1698-4eaa-9238-0c9313875664" path="/var/lib/kubelet/pods/8cc28bb0-1698-4eaa-9238-0c9313875664/volumes" Nov 03 23:07:38.407815 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:38.407767 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt"] Nov 03 23:07:38.408029 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:38.408003 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" containerID="cri-o://b3314333f7f73292a97824e49a536e040ec329e998a7168cb9bb9840ac0af972" gracePeriod=30 Nov 03 23:07:40.205413 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:40.205379 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:07:40.785239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:40.785196 5272 patch_prober.go:29] interesting pod/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:07:40.785411 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:40.785270 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:07:42.280465 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.280435 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9"] Nov 03 23:07:42.280946 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.280930 5272 cpu_manager.go:414] "containerMap: removing container" podUID="54c590d3-3731-46ad-8c50-18d4a132f5c9" containerName="revoke-keys" Nov 03 23:07:42.281003 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.280947 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c590d3-3731-46ad-8c50-18d4a132f5c9" containerName="revoke-keys" Nov 03 23:07:42.281003 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.280978 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8cc28bb0-1698-4eaa-9238-0c9313875664" containerName="payload-pre-processing" Nov 03 23:07:42.281003 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.280986 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc28bb0-1698-4eaa-9238-0c9313875664" containerName="payload-pre-processing" Nov 03 23:07:42.281108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.281050 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc28bb0-1698-4eaa-9238-0c9313875664" containerName="payload-pre-processing" Nov 03 23:07:42.281108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.281066 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c590d3-3731-46ad-8c50-18d4a132f5c9" containerName="revoke-keys" Nov 03 23:07:42.284291 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.284271 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.380383 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.380341 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rznv\" (UniqueName: \"kubernetes.io/projected/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-kube-api-access-4rznv\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.380562 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.380538 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.380621 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.380566 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.381139 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.381122 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:42.381272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.381255 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:07:42.381344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.381325 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:07:42.383480 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.383457 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9"] Nov 03 23:07:42.384095 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.384079 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:07:42.385651 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.385628 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5"] Nov 03 23:07:42.389193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.389171 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z"] Nov 03 23:07:42.389328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.389307 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.433983 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.433964 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5"] Nov 03 23:07:42.481029 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.481004 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbzd\" (UniqueName: \"kubernetes.io/projected/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-kube-api-access-nrbzd\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.481154 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.481050 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.481246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.481224 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.481311 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.481266 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.481365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.481312 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rznv\" (UniqueName: \"kubernetes.io/projected/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-kube-api-access-4rznv\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.481420 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.481381 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.481945 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.481922 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.483473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.483455 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.488098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.488076 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw"] Nov 03 23:07:42.539433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.539406 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rznv\" (UniqueName: \"kubernetes.io/projected/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-kube-api-access-4rznv\") pod \"maas-api-revoke-sub-7e0025c2-dgdd9\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.582859 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.582830 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.583333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.583307 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.583451 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.583405 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbzd\" (UniqueName: \"kubernetes.io/projected/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-kube-api-access-nrbzd\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.583936 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.583918 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.585081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.585060 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.597123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.597103 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:42.641255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.641226 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbzd\" (UniqueName: \"kubernetes.io/projected/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-kube-api-access-nrbzd\") pod \"maas-api-revoke-sub-7e1a6d2b-p8sx5\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.703203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.703173 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:42.967919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:42.967841 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9"] Nov 03 23:07:42.974059 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:07:42.974032 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a9b2f3_51f3_4d14_b548_8a030b96f6b4.slice/crio-6b7ab8e51381305a74117b4f236ddc6cd9d92c835097199f868f77a3fbef440d WatchSource:0}: Error finding container 6b7ab8e51381305a74117b4f236ddc6cd9d92c835097199f868f77a3fbef440d: Status 404 returned error can't find the container with id 6b7ab8e51381305a74117b4f236ddc6cd9d92c835097199f868f77a3fbef440d Nov 03 23:07:43.073638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.073611 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5"] Nov 03 23:07:43.080280 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:07:43.080252 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b46c2c_8ad7_4a93_a377_7e8c2e6f2724.slice/crio-97ae198885e662d63b2b205950b77a06468719de4418f9722e2af04f834a4545 WatchSource:0}: Error finding container 97ae198885e662d63b2b205950b77a06468719de4418f9722e2af04f834a4545: Status 404 returned error can't find the container with id 97ae198885e662d63b2b205950b77a06468719de4418f9722e2af04f834a4545 Nov 03 23:07:43.368521 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.368490 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" event={"ID":"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724","Type":"ContainerStarted","Data":"8c0d8d79143989c4508bcecbe26fd08ec634eb923772ee93b1c9fd2c230d09ec"} Nov 03 23:07:43.368917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.368543 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" event={"ID":"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724","Type":"ContainerStarted","Data":"97ae198885e662d63b2b205950b77a06468719de4418f9722e2af04f834a4545"} Nov 03 23:07:43.370132 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.370102 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" event={"ID":"54a9b2f3-51f3-4d14-b548-8a030b96f6b4","Type":"ContainerStarted","Data":"117642e31b6afcf109636fda7fac6abcbd5787f4389a578e78d13307a97a6f8c"} Nov 03 23:07:43.370253 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.370150 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" event={"ID":"54a9b2f3-51f3-4d14-b548-8a030b96f6b4","Type":"ContainerStarted","Data":"6b7ab8e51381305a74117b4f236ddc6cd9d92c835097199f868f77a3fbef440d"} Nov 03 23:07:43.591092 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.591041 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" podStartSLOduration=1.591025771 podStartE2EDuration="1.591025771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:43.590418934 +0000 UTC m=+2644.212727242" watchObservedRunningTime="2026-11-03 23:07:43.591025771 +0000 UTC m=+2644.213334077" Nov 03 23:07:43.592519 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.592480 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" podStartSLOduration=1.592469088 podStartE2EDuration="1.592469088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:43.480013059 +0000 UTC m=+2644.102321387" watchObservedRunningTime="2026-11-03 23:07:43.592469088 +0000 UTC m=+2644.214777423" Nov 03 23:07:43.648362 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.648331 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 23:07:43.697475 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697443 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-envoy\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.697642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697516 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-kube-api-access-kq8k4\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-kube-api-access-kq8k4\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.697642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697556 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-istio-token\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-istio-token\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.697642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697612 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-data\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.697844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697652 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-certs\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.697844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697736 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-credential-socket\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.697844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697778 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9829cd7a-0856-4739-a116-68de1211fb5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829cd7a-0856-4739-a116-68de1211fb5b-istiod-ca-cert\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.698028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697952 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-data" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.698028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.697959 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9829cd7a-0856-4739-a116-68de1211fb5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829cd7a-0856-4739-a116-68de1211fb5b-istio-podinfo\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.698145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698033 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-credential-socket" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.698145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698052 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-socket\") pod \"9829cd7a-0856-4739-a116-68de1211fb5b\" (UID: \"9829cd7a-0856-4739-a116-68de1211fb5b\") " Nov 03 23:07:43.698275 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698246 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-socket" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.698351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698285 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9829cd7a-0856-4739-a116-68de1211fb5b-istiod-ca-cert" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:43.698470 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698446 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.698584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698474 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829cd7a-0856-4739-a116-68de1211fb5b-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.698584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698487 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.698584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698500 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.698584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.698451 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-certs" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.699757 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.699729 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-kube-api-access-kq8k4" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "kube-api-access-kq8k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:43.699877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.699797 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9829cd7a-0856-4739-a116-68de1211fb5b-istio-podinfo" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:43.700091 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.700067 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-envoy" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.700091 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.700080 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-istio-token" pod "9829cd7a-0856-4739-a116-68de1211fb5b" (UID: "9829cd7a-0856-4739-a116-68de1211fb5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:43.721869 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.721841 5272 scope.go:122] "RemoveContainer" containerID="e65c733b6d2f8086fd4570e5ceb0fdf09ed7177b77c281f5c317b97e037ecb89" Nov 03 23:07:43.730905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.730884 5272 scope.go:122] "RemoveContainer" containerID="6f6505a27d7a08cd048ccb36209d87d46e23317790aa05ddeb4d41279477c6aa" Nov 03 23:07:43.739147 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.739131 5272 scope.go:122] "RemoveContainer" containerID="89178ca87a5c3bee115f56205134bb817613f85e82d07bf7af7adf8e338676e6" Nov 03 23:07:43.748151 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.748135 5272 scope.go:122] "RemoveContainer" containerID="09c390657ae68f5dd21665aaeadfde341fa451c08ed68ed92c08aada3ca13f7c" Nov 03 23:07:43.756751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.756723 5272 scope.go:122] "RemoveContainer" containerID="c37e1342b9894b143f3d01e6c0ff01c977d7405439efe301e5674f2debf0ebc4" Nov 03 23:07:43.765752 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.765728 5272 scope.go:122] "RemoveContainer" containerID="b3314333f7f73292a97824e49a536e040ec329e998a7168cb9bb9840ac0af972" Nov 03 23:07:43.799196 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.799173 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.799294 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.799209 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829cd7a-0856-4739-a116-68de1211fb5b-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.799294 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.799219 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829cd7a-0856-4739-a116-68de1211fb5b-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.799294 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.799227 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq8k4\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-kube-api-access-kq8k4\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.799294 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:43.799237 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829cd7a-0856-4739-a116-68de1211fb5b-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:44.378669 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.378620 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" event={"ID":"9829cd7a-0856-4739-a116-68de1211fb5b","Type":"ContainerDied","Data":"b3314333f7f73292a97824e49a536e040ec329e998a7168cb9bb9840ac0af972"} Nov 03 23:07:44.385389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.378682 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" event={"ID":"9829cd7a-0856-4739-a116-68de1211fb5b","Type":"ContainerDied","Data":"5fc894c59636d62d1c6715500034363fd86857b84d33968c0a68345b87d89573"} Nov 03 23:07:44.385389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.380479 5272 generic.go:356] "Generic (PLEG): container finished" podID="55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" containerID="8c0d8d79143989c4508bcecbe26fd08ec634eb923772ee93b1c9fd2c230d09ec" exitCode=0 Nov 03 23:07:44.385389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.380549 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" event={"ID":"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724","Type":"ContainerDied","Data":"8c0d8d79143989c4508bcecbe26fd08ec634eb923772ee93b1c9fd2c230d09ec"} Nov 03 23:07:44.385389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.381978 5272 generic.go:356] "Generic (PLEG): container finished" podID="54a9b2f3-51f3-4d14-b548-8a030b96f6b4" containerID="117642e31b6afcf109636fda7fac6abcbd5787f4389a578e78d13307a97a6f8c" exitCode=0 Nov 03 23:07:44.385389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.382019 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt" Nov 03 23:07:44.385389 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.382033 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" event={"ID":"54a9b2f3-51f3-4d14-b548-8a030b96f6b4","Type":"ContainerDied","Data":"117642e31b6afcf109636fda7fac6abcbd5787f4389a578e78d13307a97a6f8c"} Nov 03 23:07:44.665654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.665556 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt"] Nov 03 23:07:44.670717 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:44.670689 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5e72ad-openshift-default-5fbd65f49c-mvhjt"] Nov 03 23:07:45.492335 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.492305 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" path="/var/lib/kubelet/pods/9829cd7a-0856-4739-a116-68de1211fb5b/volumes" Nov 03 23:07:45.528203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.528177 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:45.563385 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.563363 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:45.616700 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.616671 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-openshift-ca\") pod \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " Nov 03 23:07:45.616904 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.616738 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-kube-api-access-4rznv\" (UniqueName: \"kubernetes.io/projected/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-kube-api-access-4rznv\") pod \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " Nov 03 23:07:45.616904 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.616877 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-xks-ca\") pod \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\" (UID: \"54a9b2f3-51f3-4d14-b548-8a030b96f6b4\") " Nov 03 23:07:45.617094 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.617072 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-openshift-ca" pod "54a9b2f3-51f3-4d14-b548-8a030b96f6b4" (UID: "54a9b2f3-51f3-4d14-b548-8a030b96f6b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:45.617248 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.617229 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:45.618911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.618884 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-xks-ca" pod "54a9b2f3-51f3-4d14-b548-8a030b96f6b4" (UID: "54a9b2f3-51f3-4d14-b548-8a030b96f6b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:45.619037 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.619017 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-kube-api-access-4rznv" pod "54a9b2f3-51f3-4d14-b548-8a030b96f6b4" (UID: "54a9b2f3-51f3-4d14-b548-8a030b96f6b4"). InnerVolumeSpecName "kube-api-access-4rznv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:45.718195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.718119 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-kube-api-access-nrbzd\" (UniqueName: \"kubernetes.io/projected/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-kube-api-access-nrbzd\") pod \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " Nov 03 23:07:45.718195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.718179 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-openshift-ca\") pod \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " Nov 03 23:07:45.718445 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.718240 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-xks-ca\") pod \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\" (UID: \"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724\") " Nov 03 23:07:45.718516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.718503 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rznv\" (UniqueName: \"kubernetes.io/projected/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-kube-api-access-4rznv\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:45.718575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.718517 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a9b2f3-51f3-4d14-b548-8a030b96f6b4-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:45.718627 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.718569 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-openshift-ca" pod "55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" (UID: "55b46c2c-8ad7-4a93-a377-7e8c2e6f2724"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:45.720130 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.720109 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-xks-ca" pod "55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" (UID: "55b46c2c-8ad7-4a93-a377-7e8c2e6f2724"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:45.720194 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.720168 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-kube-api-access-nrbzd" pod "55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" (UID: "55b46c2c-8ad7-4a93-a377-7e8c2e6f2724"). InnerVolumeSpecName "kube-api-access-nrbzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:45.819862 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.819828 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrbzd\" (UniqueName: \"kubernetes.io/projected/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-kube-api-access-nrbzd\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:45.819862 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.819856 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:45.819862 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:45.819866 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:46.395269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:46.395242 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" Nov 03 23:07:46.395458 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:46.395241 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5" event={"ID":"55b46c2c-8ad7-4a93-a377-7e8c2e6f2724","Type":"ContainerDied","Data":"97ae198885e662d63b2b205950b77a06468719de4418f9722e2af04f834a4545"} Nov 03 23:07:46.395458 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:46.395365 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ae198885e662d63b2b205950b77a06468719de4418f9722e2af04f834a4545" Nov 03 23:07:46.396679 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:46.396657 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" Nov 03 23:07:46.396828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:46.396702 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9" event={"ID":"54a9b2f3-51f3-4d14-b548-8a030b96f6b4","Type":"ContainerDied","Data":"6b7ab8e51381305a74117b4f236ddc6cd9d92c835097199f868f77a3fbef440d"} Nov 03 23:07:46.396828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:46.396748 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b7ab8e51381305a74117b4f236ddc6cd9d92c835097199f868f77a3fbef440d" Nov 03 23:07:47.177366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.177333 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf"] Nov 03 23:07:47.177860 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.177570 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" containerID="cri-o://72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36" gracePeriod=30 Nov 03 23:07:47.389854 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.389781 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" podUID="57cb4792-4340-4f77-995a-8eebd5c02792" containerName="payload-processing" containerID="cri-o://ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4" gracePeriod=25 Nov 03 23:07:47.488759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.488676 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" podUID="2ad38c13-2076-468f-ac66-85718c705402" containerName="payload-pre-processing" containerID="cri-o://ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d" gracePeriod=25 Nov 03 23:07:47.634520 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.634495 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 23:07:47.738939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.738851 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57cb4792-4340-4f77-995a-8eebd5c02792-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb4792-4340-4f77-995a-8eebd5c02792-plugins-config-volume\") pod \"57cb4792-4340-4f77-995a-8eebd5c02792\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " Nov 03 23:07:47.738939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.738916 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cb4792-4340-4f77-995a-8eebd5c02792-kube-api-access-kztbx\" (UniqueName: \"kubernetes.io/projected/57cb4792-4340-4f77-995a-8eebd5c02792-kube-api-access-kztbx\") pod \"57cb4792-4340-4f77-995a-8eebd5c02792\" (UID: \"57cb4792-4340-4f77-995a-8eebd5c02792\") " Nov 03 23:07:47.739313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.739286 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cb4792-4340-4f77-995a-8eebd5c02792-plugins-config-volume" pod "57cb4792-4340-4f77-995a-8eebd5c02792" (UID: "57cb4792-4340-4f77-995a-8eebd5c02792"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:47.741056 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.741037 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cb4792-4340-4f77-995a-8eebd5c02792-kube-api-access-kztbx" pod "57cb4792-4340-4f77-995a-8eebd5c02792" (UID: "57cb4792-4340-4f77-995a-8eebd5c02792"). InnerVolumeSpecName "kube-api-access-kztbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:47.840435 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.840394 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kztbx\" (UniqueName: \"kubernetes.io/projected/57cb4792-4340-4f77-995a-8eebd5c02792-kube-api-access-kztbx\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:47.840435 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:47.840428 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cb4792-4340-4f77-995a-8eebd5c02792-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:48.409956 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.409921 5272 generic.go:356] "Generic (PLEG): container finished" podID="57cb4792-4340-4f77-995a-8eebd5c02792" containerID="ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4" exitCode=0 Nov 03 23:07:48.410357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.409972 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" Nov 03 23:07:48.410357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.410009 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" event={"ID":"57cb4792-4340-4f77-995a-8eebd5c02792","Type":"ContainerDied","Data":"ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4"} Nov 03 23:07:48.410357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.410071 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z" event={"ID":"57cb4792-4340-4f77-995a-8eebd5c02792","Type":"ContainerDied","Data":"43f0a90c112c2792053446cf24a2542bd917ceb4e3ddd5a27dda53b6eefc9171"} Nov 03 23:07:48.410357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.410131 5272 scope.go:122] "RemoveContainer" containerID="ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4" Nov 03 23:07:48.420240 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.420219 5272 scope.go:122] "RemoveContainer" containerID="ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4" Nov 03 23:07:48.420515 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:48.420494 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4\": container with ID starting with ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4 not found: ID does not exist" containerID="ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4" Nov 03 23:07:48.420570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.420525 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4"} err="failed to get container status \"ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4\": rpc error: code = NotFound desc = could not find container \"ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4\": container with ID starting with ea39257bab737805ffbf3d05e457c92faf97abc8499167d6049aa1a224cb23e4 not found: ID does not exist" Nov 03 23:07:48.564630 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.564599 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z"] Nov 03 23:07:48.610964 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:48.610937 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-68fbd1-6d979f99cb-65n5z"] Nov 03 23:07:49.492536 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:49.492505 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cb4792-4340-4f77-995a-8eebd5c02792" path="/var/lib/kubelet/pods/57cb4792-4340-4f77-995a-8eebd5c02792/volumes" Nov 03 23:07:50.397548 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.397505 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" containerID="cri-o://b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d" gracePeriod=45 Nov 03 23:07:50.731326 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.730998 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 23:07:50.874948 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.874915 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-dshm\") pod \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " Nov 03 23:07:50.875150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875017 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-home\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-home\") pod \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " Nov 03 23:07:50.875150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875074 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-model-cache\") pod \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " Nov 03 23:07:50.875150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875117 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tls-certs\") pod \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " Nov 03 23:07:50.875346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875156 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-kube-api-access-gmnnk\" (UniqueName: \"kubernetes.io/projected/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-kube-api-access-gmnnk\") pod \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " Nov 03 23:07:50.875346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875243 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tmp-dir\") pod \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\" (UID: \"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0\") " Nov 03 23:07:50.875346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875270 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-home" pod "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" (UID: "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.875524 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875349 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-model-cache" pod "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" (UID: "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.875524 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875470 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tmp-dir" pod "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" (UID: "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.875635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875549 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:50.875635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875564 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:50.875635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.875577 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:50.877549 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.877515 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-dshm" pod "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" (UID: "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.877888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.877868 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tls-certs" pod "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" (UID: "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:50.877978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.877962 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-kube-api-access-gmnnk" pod "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" (UID: "6feb3c85-e3ef-450c-bf49-6642e0e4ddb0"). InnerVolumeSpecName "kube-api-access-gmnnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:50.977232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.977148 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmnnk\" (UniqueName: \"kubernetes.io/projected/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-kube-api-access-gmnnk\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:50.977232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.977181 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:50.977232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:50.977192 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:51.432306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.432266 5272 generic.go:356] "Generic (PLEG): container finished" podID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerID="b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d" exitCode=0 Nov 03 23:07:51.432454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.432306 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" Nov 03 23:07:51.432454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.432343 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" event={"ID":"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0","Type":"ContainerDied","Data":"b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d"} Nov 03 23:07:51.432454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.432394 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh" event={"ID":"6feb3c85-e3ef-450c-bf49-6642e0e4ddb0","Type":"ContainerDied","Data":"ef5fae090b9acdee2249665f64f40716eff1b4ff8027c1a40306ef3a8d14f058"} Nov 03 23:07:51.432454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.432411 5272 scope.go:122] "RemoveContainer" containerID="b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d" Nov 03 23:07:51.442951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.442871 5272 scope.go:122] "RemoveContainer" containerID="b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d" Nov 03 23:07:51.443191 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:51.443161 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d\": container with ID starting with b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d not found: ID does not exist" containerID="b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d" Nov 03 23:07:51.443260 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.443201 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d"} err="failed to get container status \"b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d\": rpc error: code = NotFound desc = could not find container \"b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d\": container with ID starting with b4af7dd4731a824a1f517a66b13c88bc3b5119115cc369710b0de1b6c8ddcd1d not found: ID does not exist" Nov 03 23:07:51.587515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.587478 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh"] Nov 03 23:07:51.632472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:51.632431 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-68fbd1/distinct-w0-68fbd1-kserve-77855d978d-4zsbh"] Nov 03 23:07:52.426764 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.426737 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 23:07:52.441721 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.441690 5272 generic.go:356] "Generic (PLEG): container finished" podID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerID="72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36" exitCode=0 Nov 03 23:07:52.441876 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.441745 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" Nov 03 23:07:52.441876 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.441754 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" event={"ID":"9cf16948-4bf8-472b-b64d-e091ef5f47a4","Type":"ContainerDied","Data":"72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36"} Nov 03 23:07:52.441876 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.441869 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf" event={"ID":"9cf16948-4bf8-472b-b64d-e091ef5f47a4","Type":"ContainerDied","Data":"30b63954e938e524be87b5d03ec08f5e6ed582f0f9b70421fd5ae0d5ece485f5"} Nov 03 23:07:52.442009 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.441903 5272 scope.go:122] "RemoveContainer" containerID="72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36" Nov 03 23:07:52.450942 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.450921 5272 scope.go:122] "RemoveContainer" containerID="72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36" Nov 03 23:07:52.451228 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:52.451207 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36\": container with ID starting with 72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36 not found: ID does not exist" containerID="72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36" Nov 03 23:07:52.451289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.451238 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36"} err="failed to get container status \"72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36\": rpc error: code = NotFound desc = could not find container \"72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36\": container with ID starting with 72ce2a7fe47267d9bcb31077f271a837775ee24d77170f115fc6d22bd56a5b36 not found: ID does not exist" Nov 03 23:07:52.592637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592600 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-kube-api-access-7ncbc\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-kube-api-access-7ncbc\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.592861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592669 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-certs\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.592861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592717 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-envoy\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.592861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592784 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-socket\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.592861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592837 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-credential-socket\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.593118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592880 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-data\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.593118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592928 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-podinfo\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.593118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.592994 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istiod-ca-cert\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.593118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.593041 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-token\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-token\") pod \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\" (UID: \"9cf16948-4bf8-472b-b64d-e091ef5f47a4\") " Nov 03 23:07:52.594008 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.593979 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-socket" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:52.594549 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.594457 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-data" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:52.594962 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.594941 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-credential-socket" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:52.595398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.595378 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istiod-ca-cert" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:52.595959 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.595938 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-certs" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:52.598116 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.598081 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-token" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:52.598198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.598119 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-kube-api-access-7ncbc" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "kube-api-access-7ncbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:52.598198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.598176 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-podinfo" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:52.598412 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.598392 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-envoy" pod "9cf16948-4bf8-472b-b64d-e091ef5f47a4" (UID: "9cf16948-4bf8-472b-b64d-e091ef5f47a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:52.694823 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694768 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ncbc\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-kube-api-access-7ncbc\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.694823 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694815 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.695032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694830 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.695032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694843 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.695032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694856 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.695032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694868 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.695032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694881 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.695032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694889 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.695032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.694897 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf16948-4bf8-472b-b64d-e091ef5f47a4-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.859318 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.859281 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf"] Nov 03 23:07:52.861766 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:52.861734 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ndnzf"] Nov 03 23:07:53.491642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:53.491609 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" path="/var/lib/kubelet/pods/6feb3c85-e3ef-450c-bf49-6642e0e4ddb0/volumes" Nov 03 23:07:53.492289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:53.492093 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" path="/var/lib/kubelet/pods/9cf16948-4bf8-472b-b64d-e091ef5f47a4/volumes" Nov 03 23:07:54.500183 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:07:54.500147 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:07:54.872285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:07:54.872249 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" podUID="2ad38c13-2076-468f-ac66-85718c705402" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:08:04.872841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:04.872774 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" podUID="2ad38c13-2076-468f-ac66-85718c705402" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:08:07.747660 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:07.747604 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 23:08:07.835965 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:07.835936 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad38c13-2076-468f-ac66-85718c705402-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad38c13-2076-468f-ac66-85718c705402-plugins-config-volume\") pod \"2ad38c13-2076-468f-ac66-85718c705402\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " Nov 03 23:08:07.836105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:07.835999 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad38c13-2076-468f-ac66-85718c705402-kube-api-access-nxzbv\" (UniqueName: \"kubernetes.io/projected/2ad38c13-2076-468f-ac66-85718c705402-kube-api-access-nxzbv\") pod \"2ad38c13-2076-468f-ac66-85718c705402\" (UID: \"2ad38c13-2076-468f-ac66-85718c705402\") " Nov 03 23:08:07.836304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:07.836272 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad38c13-2076-468f-ac66-85718c705402-plugins-config-volume" pod "2ad38c13-2076-468f-ac66-85718c705402" (UID: "2ad38c13-2076-468f-ac66-85718c705402"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:07.838247 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:07.838225 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad38c13-2076-468f-ac66-85718c705402-kube-api-access-nxzbv" pod "2ad38c13-2076-468f-ac66-85718c705402" (UID: "2ad38c13-2076-468f-ac66-85718c705402"). InnerVolumeSpecName "kube-api-access-nxzbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:07.937640 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:07.937606 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad38c13-2076-468f-ac66-85718c705402-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:07.937640 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:07.937632 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxzbv\" (UniqueName: \"kubernetes.io/projected/2ad38c13-2076-468f-ac66-85718c705402-kube-api-access-nxzbv\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:08.540007 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.539966 5272 generic.go:356] "Generic (PLEG): container finished" podID="2ad38c13-2076-468f-ac66-85718c705402" containerID="ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d" exitCode=0 Nov 03 23:08:08.540193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.540012 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" Nov 03 23:08:08.540193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.540046 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" event={"ID":"2ad38c13-2076-468f-ac66-85718c705402","Type":"ContainerDied","Data":"ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d"} Nov 03 23:08:08.540193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.540097 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw" event={"ID":"2ad38c13-2076-468f-ac66-85718c705402","Type":"ContainerDied","Data":"c62e30e8d79cab70a3b0f5189cacc835e4e9447b640ad9db866d3a18a89751b8"} Nov 03 23:08:08.540193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.540118 5272 scope.go:122] "RemoveContainer" containerID="ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d" Nov 03 23:08:08.549648 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.549633 5272 scope.go:122] "RemoveContainer" containerID="ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d" Nov 03 23:08:08.549934 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:08:08.549916 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d\": container with ID starting with ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d not found: ID does not exist" containerID="ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d" Nov 03 23:08:08.549979 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.549941 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d"} err="failed to get container status \"ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d\": rpc error: code = NotFound desc = could not find container \"ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d\": container with ID starting with ecc444573d2b746cb4fb22d6e15e0a8490cb39bca3f99ed597dbeac9d0d8c85d not found: ID does not exist" Nov 03 23:08:08.693675 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.693647 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw"] Nov 03 23:08:08.699424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.699398 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-68fbd1-76d7f5c887-p9lsw"] Nov 03 23:08:08.889881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.889787 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp"] Nov 03 23:08:08.897121 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:08.897086 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-68fbd1-t4cwp"] Nov 03 23:08:09.492204 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:09.492173 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad38c13-2076-468f-ac66-85718c705402" path="/var/lib/kubelet/pods/2ad38c13-2076-468f-ac66-85718c705402/volumes" Nov 03 23:08:09.492609 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:09.492594 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c590d3-3731-46ad-8c50-18d4a132f5c9" path="/var/lib/kubelet/pods/54c590d3-3731-46ad-8c50-18d4a132f5c9/volumes" Nov 03 23:08:09.560580 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:09.560548 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:08:24.500827 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:08:24.500755 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:08:31.047474 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:31.047432 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf"] Nov 03 23:08:31.094175 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:31.094140 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn"] Nov 03 23:08:31.095795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:31.095775 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728a5e13-zmkrf"] Nov 03 23:08:31.097308 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:31.097284 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b61084-kq4pn"] Nov 03 23:08:31.491511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:31.491417 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18342e26-c736-4b45-b358-aa5f9e1ed575" path="/var/lib/kubelet/pods/18342e26-c736-4b45-b358-aa5f9e1ed575/volumes" Nov 03 23:08:31.491975 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:31.491954 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbe10fe-38b5-4888-a5c6-759ba0006c63" path="/var/lib/kubelet/pods/9fbe10fe-38b5-4888-a5c6-759ba0006c63/volumes" Nov 03 23:08:32.039081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:32.039050 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:08:43.856422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:43.856386 5272 scope.go:122] "RemoveContainer" containerID="55b867b0929401184ccfa846382acc5ccceaaecd9b7d727fb2b187cc2f90ef26" Nov 03 23:08:43.865483 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:43.865457 5272 scope.go:122] "RemoveContainer" containerID="0ac5a69b19f9387740b8f517a3c702a64bf6eee5bbdf3b293bb7959f5181fe4e" Nov 03 23:08:43.874253 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:43.874232 5272 scope.go:122] "RemoveContainer" containerID="a8573513f4c82486e5bcf8dd9841764448507392e6c714023f75ff2f05b32005" Nov 03 23:08:43.882974 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:43.882945 5272 scope.go:122] "RemoveContainer" containerID="9c49bdedca283344ee99c4f028528c431e5ca9f77a88c4139aaa8bce31e0a035" Nov 03 23:08:43.891987 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:43.891967 5272 scope.go:122] "RemoveContainer" containerID="d86e19123f70d1c1abb640f2f1481d39549fa804582029e1133db0fc45653417" Nov 03 23:08:47.645908 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:47.645869 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp"] Nov 03 23:08:47.646383 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:47.646066 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" containerID="cri-o://a39407dfa7291dc2cc6b0286e871d6bd7374b724a8b5b5d784fd3ce11deaac25" gracePeriod=30 Nov 03 23:08:48.639088 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:48.639049 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xmfvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:08:48.639276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:48.639126 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:08:52.798306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.798267 5272 generic.go:356] "Generic (PLEG): container finished" podID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerID="a39407dfa7291dc2cc6b0286e871d6bd7374b724a8b5b5d784fd3ce11deaac25" exitCode=0 Nov 03 23:08:52.798721 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.798342 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" event={"ID":"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7","Type":"ContainerDied","Data":"a39407dfa7291dc2cc6b0286e871d6bd7374b724a8b5b5d784fd3ce11deaac25"} Nov 03 23:08:52.892251 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.892224 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 23:08:52.994123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994039 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-podinfo\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994111 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-data\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994143 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-kube-api-access-9d4qf\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-kube-api-access-9d4qf\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994196 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-certs\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994245 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-socket\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994276 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istiod-ca-cert\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994365 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-envoy\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994390 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-credential-socket\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994433 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-data" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:52.994606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994515 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-socket" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:52.994606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994526 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-certs" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:52.994606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994552 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-token\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-token\") pod \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\" (UID: \"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7\") " Nov 03 23:08:52.994983 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994765 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-credential-socket" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:52.994983 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.994767 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istiod-ca-cert" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:52.995104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.995015 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:52.995104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.995036 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:52.995104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.995051 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:52.995104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.995065 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:52.995104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.995078 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:52.996458 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.996431 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-podinfo" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:08:52.996601 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.996582 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-kube-api-access-9d4qf" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "kube-api-access-9d4qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:52.996650 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.996601 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-envoy" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:52.996687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:52.996667 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-token" pod "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" (UID: "ce1b588c-ba6c-4546-bd6f-b71d7efacdc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:53.096394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.096354 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:53.096394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.096392 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:53.096604 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.096404 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:53.096604 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.096414 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4qf\" (UniqueName: \"kubernetes.io/projected/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7-kube-api-access-9d4qf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:08:53.805902 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.805866 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" Nov 03 23:08:53.805902 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.805879 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp" event={"ID":"ce1b588c-ba6c-4546-bd6f-b71d7efacdc7","Type":"ContainerDied","Data":"9c71df26bbf4911454740020987e0e8145eda1d09a582f1d6c1f309eeaa5f1a2"} Nov 03 23:08:53.806414 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.805948 5272 scope.go:122] "RemoveContainer" containerID="a39407dfa7291dc2cc6b0286e871d6bd7374b724a8b5b5d784fd3ce11deaac25" Nov 03 23:08:53.959856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:53.959821 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp"] Nov 03 23:08:54.006597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:54.006557 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xmfvp"] Nov 03 23:08:54.499686 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:08:54.499645 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:08:55.492483 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:55.492453 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" path="/var/lib/kubelet/pods/ce1b588c-ba6c-4546-bd6f-b71d7efacdc7/volumes" Nov 03 23:08:58.525041 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525003 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v"] Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525405 5272 cpu_manager.go:414] "containerMap: removing container" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525416 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525428 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2ad38c13-2076-468f-ac66-85718c705402" containerName="payload-pre-processing" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525433 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad38c13-2076-468f-ac66-85718c705402" containerName="payload-pre-processing" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525440 5272 cpu_manager.go:414] "containerMap: removing container" podUID="57cb4792-4340-4f77-995a-8eebd5c02792" containerName="payload-processing" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525446 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cb4792-4340-4f77-995a-8eebd5c02792" containerName="payload-processing" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525460 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525465 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525471 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525475 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525483 5272 cpu_manager.go:414] "containerMap: removing container" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" Nov 03 23:08:58.525482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525489 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525503 5272 cpu_manager.go:414] "containerMap: removing container" podUID="54a9b2f3-51f3-4d14-b548-8a030b96f6b4" containerName="revoke-keys" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525508 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a9b2f3-51f3-4d14-b548-8a030b96f6b4" containerName="revoke-keys" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525515 5272 cpu_manager.go:414] "containerMap: removing container" podUID="55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" containerName="revoke-keys" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525520 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" containerName="revoke-keys" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525584 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf16948-4bf8-472b-b64d-e091ef5f47a4" containerName="istio-proxy" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525593 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1b588c-ba6c-4546-bd6f-b71d7efacdc7" containerName="istio-proxy" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525599 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="9829cd7a-0856-4739-a116-68de1211fb5b" containerName="istio-proxy" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525607 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a9b2f3-51f3-4d14-b548-8a030b96f6b4" containerName="revoke-keys" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525613 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" containerName="revoke-keys" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525622 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="6feb3c85-e3ef-450c-bf49-6642e0e4ddb0" containerName="main" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525628 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad38c13-2076-468f-ac66-85718c705402" containerName="payload-pre-processing" Nov 03 23:08:58.525960 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.525634 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cb4792-4340-4f77-995a-8eebd5c02792" containerName="payload-processing" Nov 03 23:08:58.530335 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.530313 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.589765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.589735 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:08:58.636198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.636164 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:08:58.636198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.636190 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:08:58.636422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.636166 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:08:58.638260 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.638239 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v"] Nov 03 23:08:58.654650 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.654617 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.654759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.654655 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.654838 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.654751 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szv89\" (UniqueName: \"kubernetes.io/projected/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-kube-api-access-szv89\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.755521 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.755488 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.755718 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.755542 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szv89\" (UniqueName: \"kubernetes.io/projected/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-kube-api-access-szv89\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.755718 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.755642 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.756256 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.756226 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.757809 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.757782 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.813114 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.813083 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szv89\" (UniqueName: \"kubernetes.io/projected/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-kube-api-access-szv89\") pod \"maas-api-revoke-sub-8eb4c8ef-lxr6v\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:58.843184 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:58.843153 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:08:59.216881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:59.216766 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v"] Nov 03 23:08:59.223338 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:08:59.223310 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb9a729_bf6f_4978_bf3b_a1bac42b9efe.slice/crio-d464242dce20f016da2497e6a013190da097cbc2efc8ddb92ded28a5468b8ed1 WatchSource:0}: Error finding container d464242dce20f016da2497e6a013190da097cbc2efc8ddb92ded28a5468b8ed1: Status 404 returned error can't find the container with id d464242dce20f016da2497e6a013190da097cbc2efc8ddb92ded28a5468b8ed1 Nov 03 23:08:59.842660 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:59.842570 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" event={"ID":"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe","Type":"ContainerStarted","Data":"1c3db130f126db6b4635700d866a4dbdf6f89c071358825b469cbbe7b2fcfd5e"} Nov 03 23:08:59.842660 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:59.842652 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" event={"ID":"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe","Type":"ContainerStarted","Data":"d464242dce20f016da2497e6a013190da097cbc2efc8ddb92ded28a5468b8ed1"} Nov 03 23:08:59.951759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:08:59.951703 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" podStartSLOduration=1.951689241 podStartE2EDuration="1.951689241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:59.950369708 +0000 UTC m=+2720.572678017" watchObservedRunningTime="2026-11-03 23:08:59.951689241 +0000 UTC m=+2720.573997548" Nov 03 23:09:00.849343 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:00.849300 5272 generic.go:356] "Generic (PLEG): container finished" podID="8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" containerID="1c3db130f126db6b4635700d866a4dbdf6f89c071358825b469cbbe7b2fcfd5e" exitCode=0 Nov 03 23:09:00.849798 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:00.849399 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" event={"ID":"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe","Type":"ContainerDied","Data":"1c3db130f126db6b4635700d866a4dbdf6f89c071358825b469cbbe7b2fcfd5e"} Nov 03 23:09:02.002169 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.002134 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:09:02.088313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.088274 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-kube-api-access-szv89\" (UniqueName: \"kubernetes.io/projected/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-kube-api-access-szv89\") pod \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " Nov 03 23:09:02.088508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.088368 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-xks-ca\") pod \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " Nov 03 23:09:02.088508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.088422 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-openshift-ca\") pod \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\" (UID: \"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe\") " Nov 03 23:09:02.088785 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.088752 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-openshift-ca" pod "8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" (UID: "8bb9a729-bf6f-4978-bf3b-a1bac42b9efe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:02.088931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.088897 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:02.090470 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.090442 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-kube-api-access-szv89" pod "8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" (UID: "8bb9a729-bf6f-4978-bf3b-a1bac42b9efe"). InnerVolumeSpecName "kube-api-access-szv89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:02.090925 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.090894 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-xks-ca" pod "8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" (UID: "8bb9a729-bf6f-4978-bf3b-a1bac42b9efe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:02.190012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.189924 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szv89\" (UniqueName: \"kubernetes.io/projected/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-kube-api-access-szv89\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:02.190012 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.189957 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:02.871825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.871510 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" event={"ID":"8bb9a729-bf6f-4978-bf3b-a1bac42b9efe","Type":"ContainerDied","Data":"d464242dce20f016da2497e6a013190da097cbc2efc8ddb92ded28a5468b8ed1"} Nov 03 23:09:02.871825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.871640 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d464242dce20f016da2497e6a013190da097cbc2efc8ddb92ded28a5468b8ed1" Nov 03 23:09:02.871825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:02.871734 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v" Nov 03 23:09:11.519240 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.519201 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp"] Nov 03 23:09:11.519648 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.519594 5272 cpu_manager.go:414] "containerMap: removing container" podUID="8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" containerName="revoke-keys" Nov 03 23:09:11.519648 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.519605 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" containerName="revoke-keys" Nov 03 23:09:11.519721 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.519700 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" containerName="revoke-keys" Nov 03 23:09:11.522917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.522894 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.571518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.571477 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:09:11.608325 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.608297 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:09:11.617858 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.617832 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:09:11.617858 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.617845 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:09:11.619867 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.619841 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp"] Nov 03 23:09:11.693560 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.693520 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.693560 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.693559 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.693794 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.693588 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwdg\" (UniqueName: \"kubernetes.io/projected/796a1c3a-54ab-4f70-8574-c499b89f0d24-kube-api-access-ddwdg\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.794517 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.794477 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.794731 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.794525 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.794731 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.794561 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwdg\" (UniqueName: \"kubernetes.io/projected/796a1c3a-54ab-4f70-8574-c499b89f0d24-kube-api-access-ddwdg\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.795344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.795321 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.797030 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.797005 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:11.852384 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:11.852343 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwdg\" (UniqueName: \"kubernetes.io/projected/796a1c3a-54ab-4f70-8574-c499b89f0d24-kube-api-access-ddwdg\") pod \"maas-api-revoke-sub-bcc98f9c-hh7jp\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:12.135927 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:12.135846 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:12.506231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:12.506142 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp"] Nov 03 23:09:12.513403 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:09:12.513375 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796a1c3a_54ab_4f70_8574_c499b89f0d24.slice/crio-949349e7b70c2ecf55f71748e43e6c88a211ed3bf8626d8c2ed82251fa124509 WatchSource:0}: Error finding container 949349e7b70c2ecf55f71748e43e6c88a211ed3bf8626d8c2ed82251fa124509: Status 404 returned error can't find the container with id 949349e7b70c2ecf55f71748e43e6c88a211ed3bf8626d8c2ed82251fa124509 Nov 03 23:09:12.936056 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:12.936007 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" event={"ID":"796a1c3a-54ab-4f70-8574-c499b89f0d24","Type":"ContainerStarted","Data":"c8841f7031d9b88b1069f5d51a329f18bf4766510bc115c105faebcb0f2c33ca"} Nov 03 23:09:12.936462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:12.936077 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" event={"ID":"796a1c3a-54ab-4f70-8574-c499b89f0d24","Type":"ContainerStarted","Data":"949349e7b70c2ecf55f71748e43e6c88a211ed3bf8626d8c2ed82251fa124509"} Nov 03 23:09:13.045570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:13.045515 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" podStartSLOduration=2.045499287 podStartE2EDuration="2.045499287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:13.04382219 +0000 UTC m=+2733.666130493" watchObservedRunningTime="2026-11-03 23:09:13.045499287 +0000 UTC m=+2733.667807594" Nov 03 23:09:13.944452 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:13.944414 5272 generic.go:356] "Generic (PLEG): container finished" podID="796a1c3a-54ab-4f70-8574-c499b89f0d24" containerID="c8841f7031d9b88b1069f5d51a329f18bf4766510bc115c105faebcb0f2c33ca" exitCode=0 Nov 03 23:09:13.944955 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:13.944501 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" event={"ID":"796a1c3a-54ab-4f70-8574-c499b89f0d24","Type":"ContainerDied","Data":"c8841f7031d9b88b1069f5d51a329f18bf4766510bc115c105faebcb0f2c33ca"} Nov 03 23:09:15.079316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.079290 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:15.228327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.228244 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796a1c3a-54ab-4f70-8574-c499b89f0d24-kube-api-access-ddwdg\" (UniqueName: \"kubernetes.io/projected/796a1c3a-54ab-4f70-8574-c499b89f0d24-kube-api-access-ddwdg\") pod \"796a1c3a-54ab-4f70-8574-c499b89f0d24\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " Nov 03 23:09:15.228327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.228296 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-openshift-ca\") pod \"796a1c3a-54ab-4f70-8574-c499b89f0d24\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " Nov 03 23:09:15.228589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.228372 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-xks-ca\") pod \"796a1c3a-54ab-4f70-8574-c499b89f0d24\" (UID: \"796a1c3a-54ab-4f70-8574-c499b89f0d24\") " Nov 03 23:09:15.228754 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.228729 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-openshift-ca" pod "796a1c3a-54ab-4f70-8574-c499b89f0d24" (UID: "796a1c3a-54ab-4f70-8574-c499b89f0d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:15.230364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.230344 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796a1c3a-54ab-4f70-8574-c499b89f0d24-kube-api-access-ddwdg" pod "796a1c3a-54ab-4f70-8574-c499b89f0d24" (UID: "796a1c3a-54ab-4f70-8574-c499b89f0d24"). InnerVolumeSpecName "kube-api-access-ddwdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:15.230438 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.230393 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-xks-ca" pod "796a1c3a-54ab-4f70-8574-c499b89f0d24" (UID: "796a1c3a-54ab-4f70-8574-c499b89f0d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:15.329569 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.329530 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwdg\" (UniqueName: \"kubernetes.io/projected/796a1c3a-54ab-4f70-8574-c499b89f0d24-kube-api-access-ddwdg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:15.329569 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.329563 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:15.329794 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.329579 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a1c3a-54ab-4f70-8574-c499b89f0d24-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:15.957972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.957927 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" event={"ID":"796a1c3a-54ab-4f70-8574-c499b89f0d24","Type":"ContainerDied","Data":"949349e7b70c2ecf55f71748e43e6c88a211ed3bf8626d8c2ed82251fa124509"} Nov 03 23:09:15.958201 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.957977 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp" Nov 03 23:09:15.958201 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:15.958001 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949349e7b70c2ecf55f71748e43e6c88a211ed3bf8626d8c2ed82251fa124509" Nov 03 23:09:16.071671 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:16.071621 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:09:24.500544 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:09:24.500500 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:09:34.857564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:34.857525 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:09:38.415439 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:38.415401 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq"] Nov 03 23:09:38.998386 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:38.998348 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7"] Nov 03 23:09:38.998786 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:38.998760 5272 cpu_manager.go:414] "containerMap: removing container" podUID="796a1c3a-54ab-4f70-8574-c499b89f0d24" containerName="revoke-keys" Nov 03 23:09:38.998861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:38.998787 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796a1c3a-54ab-4f70-8574-c499b89f0d24" containerName="revoke-keys" Nov 03 23:09:38.998912 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:38.998901 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="796a1c3a-54ab-4f70-8574-c499b89f0d24" containerName="revoke-keys" Nov 03 23:09:39.002032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.002010 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.051206 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.051174 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:09:39.051670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.051648 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:09:39.061289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.061246 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lt2k\" (UniqueName: \"kubernetes.io/projected/b67d3af5-e348-41f4-9d7a-a44f64297798-kube-api-access-4lt2k\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.061289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.061286 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.061518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.061310 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.098221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.098188 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:09:39.098368 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.098281 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:09:39.100345 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.100327 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7"] Nov 03 23:09:39.162916 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.162881 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lt2k\" (UniqueName: \"kubernetes.io/projected/b67d3af5-e348-41f4-9d7a-a44f64297798-kube-api-access-4lt2k\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.162916 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.162917 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.163199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.162937 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.163634 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.163607 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.165293 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.165268 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.223266 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.223236 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lt2k\" (UniqueName: \"kubernetes.io/projected/b67d3af5-e348-41f4-9d7a-a44f64297798-kube-api-access-4lt2k\") pod \"maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.315041 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.315004 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:39.685777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:39.685695 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7"] Nov 03 23:09:39.693000 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:09:39.692973 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67d3af5_e348_41f4_9d7a_a44f64297798.slice/crio-72abfc130565201e768da12a385a75467b8275db8a9e323682be1097f8c34efe WatchSource:0}: Error finding container 72abfc130565201e768da12a385a75467b8275db8a9e323682be1097f8c34efe: Status 404 returned error can't find the container with id 72abfc130565201e768da12a385a75467b8275db8a9e323682be1097f8c34efe Nov 03 23:09:40.091409 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:40.091373 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" event={"ID":"b67d3af5-e348-41f4-9d7a-a44f64297798","Type":"ContainerStarted","Data":"9ef02425f21edff73f406dd0abdf1f15651274bf848e8da579afcc5bcde260ff"} Nov 03 23:09:40.091610 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:40.091427 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" event={"ID":"b67d3af5-e348-41f4-9d7a-a44f64297798","Type":"ContainerStarted","Data":"72abfc130565201e768da12a385a75467b8275db8a9e323682be1097f8c34efe"} Nov 03 23:09:40.204287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:40.204190 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" podStartSLOduration=2.204176115 podStartE2EDuration="2.204176115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:40.203395278 +0000 UTC m=+2760.825703586" watchObservedRunningTime="2026-11-03 23:09:40.204176115 +0000 UTC m=+2760.826484453" Nov 03 23:09:41.097985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:41.097945 5272 generic.go:356] "Generic (PLEG): container finished" podID="b67d3af5-e348-41f4-9d7a-a44f64297798" containerID="9ef02425f21edff73f406dd0abdf1f15651274bf848e8da579afcc5bcde260ff" exitCode=0 Nov 03 23:09:41.098413 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:41.098023 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" event={"ID":"b67d3af5-e348-41f4-9d7a-a44f64297798","Type":"ContainerDied","Data":"9ef02425f21edff73f406dd0abdf1f15651274bf848e8da579afcc5bcde260ff"} Nov 03 23:09:42.231337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.231309 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:42.294089 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.294059 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-xks-ca\") pod \"b67d3af5-e348-41f4-9d7a-a44f64297798\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " Nov 03 23:09:42.294287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.294110 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-openshift-ca\") pod \"b67d3af5-e348-41f4-9d7a-a44f64297798\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " Nov 03 23:09:42.294287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.294209 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67d3af5-e348-41f4-9d7a-a44f64297798-kube-api-access-4lt2k\" (UniqueName: \"kubernetes.io/projected/b67d3af5-e348-41f4-9d7a-a44f64297798-kube-api-access-4lt2k\") pod \"b67d3af5-e348-41f4-9d7a-a44f64297798\" (UID: \"b67d3af5-e348-41f4-9d7a-a44f64297798\") " Nov 03 23:09:42.294525 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.294502 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-openshift-ca" pod "b67d3af5-e348-41f4-9d7a-a44f64297798" (UID: "b67d3af5-e348-41f4-9d7a-a44f64297798"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:42.296208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.296183 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-xks-ca" pod "b67d3af5-e348-41f4-9d7a-a44f64297798" (UID: "b67d3af5-e348-41f4-9d7a-a44f64297798"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:42.296301 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.296280 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67d3af5-e348-41f4-9d7a-a44f64297798-kube-api-access-4lt2k" pod "b67d3af5-e348-41f4-9d7a-a44f64297798" (UID: "b67d3af5-e348-41f4-9d7a-a44f64297798"). InnerVolumeSpecName "kube-api-access-4lt2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:42.395856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.395754 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lt2k\" (UniqueName: \"kubernetes.io/projected/b67d3af5-e348-41f4-9d7a-a44f64297798-kube-api-access-4lt2k\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:42.395856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.395786 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:42.395856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:42.395796 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67d3af5-e348-41f4-9d7a-a44f64297798-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.110077 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.110047 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" Nov 03 23:09:43.110077 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.110057 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7" event={"ID":"b67d3af5-e348-41f4-9d7a-a44f64297798","Type":"ContainerDied","Data":"72abfc130565201e768da12a385a75467b8275db8a9e323682be1097f8c34efe"} Nov 03 23:09:43.110303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.110100 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72abfc130565201e768da12a385a75467b8275db8a9e323682be1097f8c34efe" Nov 03 23:09:43.948934 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.948901 5272 scope.go:122] "RemoveContainer" containerID="3be5b19594c1d0b370ef6fab680acee81b3f79cc3b4e6af343b974daeb7acb13" Nov 03 23:09:43.957855 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.957834 5272 scope.go:122] "RemoveContainer" containerID="87bfd7583663f397e289cf6b22f8fbf0820bd83777a2175cfa13f18667676f69" Nov 03 23:09:43.965897 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.965878 5272 scope.go:122] "RemoveContainer" containerID="9ef569df045b4cbb080058d7aad172a2491d2ceb828ab875b2d36791bc784fe1" Nov 03 23:09:43.973881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.973858 5272 scope.go:122] "RemoveContainer" containerID="4c96b3a826da721040a6942131fd7d1dd5eb60e16bf10e38cb6939fae4a714f0" Nov 03 23:09:43.982007 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:43.981982 5272 scope.go:122] "RemoveContainer" containerID="4098c4826f12837d0ef4773a7a195328f4d7016453023af16047cf978d36be6c" Nov 03 23:09:50.384148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:50.384111 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47"] Nov 03 23:09:50.439666 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:50.439630 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp"] Nov 03 23:09:53.435682 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.435638 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" podUID="1347c693-c8d5-4f4a-8408-9250ef8619c7" containerName="main" containerID="cri-o://c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190" gracePeriod=45 Nov 03 23:09:53.458539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.458498 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" podUID="1347c693-c8d5-4f4a-8408-9250ef8619c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.255:8000/ready\": dial tcp 10.133.0.255:8000: connect: connection refused" Nov 03 23:09:53.688744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.688684 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:09:53.792047 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.792002 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:09:53.815483 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.815446 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-dshm\") pod \"1347c693-c8d5-4f4a-8408-9250ef8619c7\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " Nov 03 23:09:53.815638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.815520 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-tmp-dir\") pod \"1347c693-c8d5-4f4a-8408-9250ef8619c7\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " Nov 03 23:09:53.815638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.815586 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1347c693-c8d5-4f4a-8408-9250ef8619c7-kube-api-access-dwtw7\" (UniqueName: \"kubernetes.io/projected/1347c693-c8d5-4f4a-8408-9250ef8619c7-kube-api-access-dwtw7\") pod \"1347c693-c8d5-4f4a-8408-9250ef8619c7\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " Nov 03 23:09:53.815785 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.815689 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1347c693-c8d5-4f4a-8408-9250ef8619c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/1347c693-c8d5-4f4a-8408-9250ef8619c7-tls-certs\") pod \"1347c693-c8d5-4f4a-8408-9250ef8619c7\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " Nov 03 23:09:53.815785 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.815750 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-home\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-home\") pod \"1347c693-c8d5-4f4a-8408-9250ef8619c7\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " Nov 03 23:09:53.815926 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.815789 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-tmp-dir" pod "1347c693-c8d5-4f4a-8408-9250ef8619c7" (UID: "1347c693-c8d5-4f4a-8408-9250ef8619c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:53.815926 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.815795 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-model-cache\") pod \"1347c693-c8d5-4f4a-8408-9250ef8619c7\" (UID: \"1347c693-c8d5-4f4a-8408-9250ef8619c7\") " Nov 03 23:09:53.816064 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.816021 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-home" pod "1347c693-c8d5-4f4a-8408-9250ef8619c7" (UID: "1347c693-c8d5-4f4a-8408-9250ef8619c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:53.816127 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.816089 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-model-cache" pod "1347c693-c8d5-4f4a-8408-9250ef8619c7" (UID: "1347c693-c8d5-4f4a-8408-9250ef8619c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:53.816225 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.816207 5272 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-home\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:53.816291 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.816225 5272 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-model-cache\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:53.816291 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.816236 5272 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-tmp-dir\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:53.817974 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.817947 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1347c693-c8d5-4f4a-8408-9250ef8619c7-kube-api-access-dwtw7" pod "1347c693-c8d5-4f4a-8408-9250ef8619c7" (UID: "1347c693-c8d5-4f4a-8408-9250ef8619c7"). InnerVolumeSpecName "kube-api-access-dwtw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:53.818112 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.818092 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1347c693-c8d5-4f4a-8408-9250ef8619c7-tls-certs" pod "1347c693-c8d5-4f4a-8408-9250ef8619c7" (UID: "1347c693-c8d5-4f4a-8408-9250ef8619c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:53.818158 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.818120 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-dshm" pod "1347c693-c8d5-4f4a-8408-9250ef8619c7" (UID: "1347c693-c8d5-4f4a-8408-9250ef8619c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:53.917348 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.917304 5272 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1347c693-c8d5-4f4a-8408-9250ef8619c7-tls-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:53.917348 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.917341 5272 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1347c693-c8d5-4f4a-8408-9250ef8619c7-dshm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:53.917348 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:53.917355 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwtw7\" (UniqueName: \"kubernetes.io/projected/1347c693-c8d5-4f4a-8408-9250ef8619c7-kube-api-access-dwtw7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:54.170473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.170436 5272 generic.go:356] "Generic (PLEG): container finished" podID="1347c693-c8d5-4f4a-8408-9250ef8619c7" containerID="c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190" exitCode=0 Nov 03 23:09:54.170691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.170480 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" Nov 03 23:09:54.170691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.170501 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" event={"ID":"1347c693-c8d5-4f4a-8408-9250ef8619c7","Type":"ContainerDied","Data":"c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190"} Nov 03 23:09:54.170691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.170541 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq" event={"ID":"1347c693-c8d5-4f4a-8408-9250ef8619c7","Type":"ContainerDied","Data":"fcd8e1fdfec62882a93901ea41f70b3a926f2b187dbb641aa1e28b89dc4ab3de"} Nov 03 23:09:54.170691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.170562 5272 scope.go:122] "RemoveContainer" containerID="c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190" Nov 03 23:09:54.180472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.180449 5272 scope.go:122] "RemoveContainer" containerID="c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190" Nov 03 23:09:54.180766 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:09:54.180747 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190\": container with ID starting with c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190 not found: ID does not exist" containerID="c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190" Nov 03 23:09:54.180886 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.180780 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190"} err="failed to get container status \"c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190\": rpc error: code = NotFound desc = could not find container \"c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190\": container with ID starting with c4e398ec9649d1b6fa393cb721212fe00f66190e25992ffc69a6e4abe886d190 not found: ID does not exist" Nov 03 23:09:54.327111 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.327077 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq"] Nov 03 23:09:54.374401 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:54.374353 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ac6633/test-model-ac6633-kserve-869955d667-zsgcq"] Nov 03 23:09:54.500191 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:09:54.500108 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:09:55.231835 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.231782 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7"] Nov 03 23:09:55.278600 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.278569 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bf80c-55xq7"] Nov 03 23:09:55.384485 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.384422 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" podUID="3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" containerName="payload-processing" containerID="cri-o://267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d" gracePeriod=25 Nov 03 23:09:55.440225 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.440178 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" podUID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" containerName="payload-pre-processing" containerID="cri-o://4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c" gracePeriod=25 Nov 03 23:09:55.493067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.492996 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1347c693-c8d5-4f4a-8408-9250ef8619c7" path="/var/lib/kubelet/pods/1347c693-c8d5-4f4a-8408-9250ef8619c7/volumes" Nov 03 23:09:55.493431 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.493418 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67d3af5-e348-41f4-9d7a-a44f64297798" path="/var/lib/kubelet/pods/b67d3af5-e348-41f4-9d7a-a44f64297798/volumes" Nov 03 23:09:55.634943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.634917 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:09:55.734754 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.734721 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-plugins-config-volume\") pod \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " Nov 03 23:09:55.734956 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.734856 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-kube-api-access-7qzlt\" (UniqueName: \"kubernetes.io/projected/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-kube-api-access-7qzlt\") pod \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\" (UID: \"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc\") " Nov 03 23:09:55.735126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.735105 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-plugins-config-volume" pod "3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" (UID: "3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:55.737005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.736977 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-kube-api-access-7qzlt" pod "3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" (UID: "3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc"). InnerVolumeSpecName "kube-api-access-7qzlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:55.835904 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.835863 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:55.835904 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:55.835897 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzlt\" (UniqueName: \"kubernetes.io/projected/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc-kube-api-access-7qzlt\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:56.190321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.190235 5272 generic.go:356] "Generic (PLEG): container finished" podID="3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" containerID="267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d" exitCode=0 Nov 03 23:09:56.190321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.190281 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" Nov 03 23:09:56.190540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.190318 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" event={"ID":"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc","Type":"ContainerDied","Data":"267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d"} Nov 03 23:09:56.190540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.190378 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47" event={"ID":"3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc","Type":"ContainerDied","Data":"ab9457529c557c2b867b09fb352d2ae21c7b07f74e08ae7f8fd39e53e51f0694"} Nov 03 23:09:56.190540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.190408 5272 scope.go:122] "RemoveContainer" containerID="267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d" Nov 03 23:09:56.200789 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.200770 5272 scope.go:122] "RemoveContainer" containerID="267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d" Nov 03 23:09:56.201098 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:09:56.201074 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d\": container with ID starting with 267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d not found: ID does not exist" containerID="267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d" Nov 03 23:09:56.201173 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.201111 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d"} err="failed to get container status \"267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d\": rpc error: code = NotFound desc = could not find container \"267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d\": container with ID starting with 267015a1fb3a740c37e717a5d5bbd75c97dd9f00f4d711f73092313080747c0d not found: ID does not exist" Nov 03 23:09:56.346038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.346008 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47"] Nov 03 23:09:56.393576 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:56.393536 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bf80c-cd68887bb-m4r47"] Nov 03 23:09:57.492079 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:09:57.492044 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" path="/var/lib/kubelet/pods/3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc/volumes" Nov 03 23:10:02.633174 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:02.633139 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" podUID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:10:11.995824 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:11.995773 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8"] Nov 03 23:10:11.996229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:11.996017 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" containerID="cri-o://0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407" gracePeriod=30 Nov 03 23:10:12.634065 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.634026 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" podUID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:10:12.872354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872318 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl"] Nov 03 23:10:12.872735 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872722 5272 cpu_manager.go:414] "containerMap: removing container" podUID="1347c693-c8d5-4f4a-8408-9250ef8619c7" containerName="main" Nov 03 23:10:12.872780 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872737 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1347c693-c8d5-4f4a-8408-9250ef8619c7" containerName="main" Nov 03 23:10:12.872780 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872760 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b67d3af5-e348-41f4-9d7a-a44f64297798" containerName="revoke-keys" Nov 03 23:10:12.872780 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872764 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67d3af5-e348-41f4-9d7a-a44f64297798" containerName="revoke-keys" Nov 03 23:10:12.872780 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872778 5272 cpu_manager.go:414] "containerMap: removing container" podUID="3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" containerName="payload-processing" Nov 03 23:10:12.872989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872783 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" containerName="payload-processing" Nov 03 23:10:12.872989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872857 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="1347c693-c8d5-4f4a-8408-9250ef8619c7" containerName="main" Nov 03 23:10:12.872989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872868 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67d3af5-e348-41f4-9d7a-a44f64297798" containerName="revoke-keys" Nov 03 23:10:12.872989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.872876 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2ed7ac-8c96-401f-aeb2-7eab48a2e1dc" containerName="payload-processing" Nov 03 23:10:12.876108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.876084 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.877591 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.877567 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.877663 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.877616 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.877735 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.877713 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwpp\" (UniqueName: \"kubernetes.io/projected/304bec98-6219-49cf-bf34-70d8e3d3c682-kube-api-access-plwpp\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.928002 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.927932 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:10:12.974782 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.974746 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:10:12.974988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.974847 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:12.974988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.974897 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:10:12.976765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.976747 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl"] Nov 03 23:10:12.978357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.978337 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.978437 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.978384 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.978437 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.978421 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwpp\" (UniqueName: \"kubernetes.io/projected/304bec98-6219-49cf-bf34-70d8e3d3c682-kube-api-access-plwpp\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.979083 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.979065 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:12.980664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:12.980643 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:13.037212 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:13.037175 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwpp\" (UniqueName: \"kubernetes.io/projected/304bec98-6219-49cf-bf34-70d8e3d3c682-kube-api-access-plwpp\") pod \"maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:13.188827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:13.188720 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:13.567843 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:13.567813 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl"] Nov 03 23:10:13.574505 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:10:13.574477 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304bec98_6219_49cf_bf34_70d8e3d3c682.slice/crio-5618f8bf618a43e4083969fa88382a6421de11f7b582de9fa71a7413fa4f7e1f WatchSource:0}: Error finding container 5618f8bf618a43e4083969fa88382a6421de11f7b582de9fa71a7413fa4f7e1f: Status 404 returned error can't find the container with id 5618f8bf618a43e4083969fa88382a6421de11f7b582de9fa71a7413fa4f7e1f Nov 03 23:10:14.300455 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:14.300415 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" event={"ID":"304bec98-6219-49cf-bf34-70d8e3d3c682","Type":"ContainerStarted","Data":"3d201221d0a0bcdf622079c4d79fcfe9e684dfbba9b19fc07f18792cfe28e597"} Nov 03 23:10:14.300928 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:14.300506 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" event={"ID":"304bec98-6219-49cf-bf34-70d8e3d3c682","Type":"ContainerStarted","Data":"5618f8bf618a43e4083969fa88382a6421de11f7b582de9fa71a7413fa4f7e1f"} Nov 03 23:10:14.410718 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:14.410668 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" podStartSLOduration=2.410652149 podStartE2EDuration="2.410652149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:10:14.409708499 +0000 UTC m=+2795.032016806" watchObservedRunningTime="2026-11-03 23:10:14.410652149 +0000 UTC m=+2795.032960456" Nov 03 23:10:15.307970 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.307935 5272 generic.go:356] "Generic (PLEG): container finished" podID="304bec98-6219-49cf-bf34-70d8e3d3c682" containerID="3d201221d0a0bcdf622079c4d79fcfe9e684dfbba9b19fc07f18792cfe28e597" exitCode=0 Nov 03 23:10:15.308398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.308002 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" event={"ID":"304bec98-6219-49cf-bf34-70d8e3d3c682","Type":"ContainerDied","Data":"3d201221d0a0bcdf622079c4d79fcfe9e684dfbba9b19fc07f18792cfe28e597"} Nov 03 23:10:15.686693 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.686666 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:10:15.706708 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.706676 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-plugins-config-volume\") pod \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " Nov 03 23:10:15.706847 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.706759 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-kube-api-access-jklkm\" (UniqueName: \"kubernetes.io/projected/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-kube-api-access-jklkm\") pod \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\" (UID: \"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5\") " Nov 03 23:10:15.707151 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.707114 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-plugins-config-volume" pod "2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" (UID: "2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:15.708836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.708811 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-kube-api-access-jklkm" pod "2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" (UID: "2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5"). InnerVolumeSpecName "kube-api-access-jklkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:15.808050 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.808011 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jklkm\" (UniqueName: \"kubernetes.io/projected/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-kube-api-access-jklkm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:15.808050 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:15.808052 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:16.314417 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.314384 5272 generic.go:356] "Generic (PLEG): container finished" podID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" containerID="4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c" exitCode=0 Nov 03 23:10:16.314912 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.314434 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" event={"ID":"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5","Type":"ContainerDied","Data":"4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c"} Nov 03 23:10:16.314912 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.314494 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" event={"ID":"2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5","Type":"ContainerDied","Data":"c4c60a122df57e826a691a07a7360ec16e9b7ac7fe49670f98cbe35751ef66c8"} Nov 03 23:10:16.314912 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.314452 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp" Nov 03 23:10:16.314912 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.314524 5272 scope.go:122] "RemoveContainer" containerID="4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c" Nov 03 23:10:16.324851 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.324794 5272 scope.go:122] "RemoveContainer" containerID="4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c" Nov 03 23:10:16.325131 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:10:16.325109 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c\": container with ID starting with 4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c not found: ID does not exist" containerID="4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c" Nov 03 23:10:16.325216 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.325145 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c"} err="failed to get container status \"4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c\": rpc error: code = NotFound desc = could not find container \"4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c\": container with ID starting with 4df965ef198d7a43822524886005cede217b1ea44c9b26adb2c4ac31ecd4e31c not found: ID does not exist" Nov 03 23:10:16.444870 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.444843 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:16.470361 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.470334 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp"] Nov 03 23:10:16.514597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.514565 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-openshift-ca\") pod \"304bec98-6219-49cf-bf34-70d8e3d3c682\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " Nov 03 23:10:16.514730 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.514606 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-xks-ca\") pod \"304bec98-6219-49cf-bf34-70d8e3d3c682\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " Nov 03 23:10:16.514730 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.514662 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304bec98-6219-49cf-bf34-70d8e3d3c682-kube-api-access-plwpp\" (UniqueName: \"kubernetes.io/projected/304bec98-6219-49cf-bf34-70d8e3d3c682-kube-api-access-plwpp\") pod \"304bec98-6219-49cf-bf34-70d8e3d3c682\" (UID: \"304bec98-6219-49cf-bf34-70d8e3d3c682\") " Nov 03 23:10:16.515038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.515008 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-openshift-ca" pod "304bec98-6219-49cf-bf34-70d8e3d3c682" (UID: "304bec98-6219-49cf-bf34-70d8e3d3c682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:16.515197 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.515106 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:16.517423 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.517390 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-xks-ca" pod "304bec98-6219-49cf-bf34-70d8e3d3c682" (UID: "304bec98-6219-49cf-bf34-70d8e3d3c682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:16.517515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.517432 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304bec98-6219-49cf-bf34-70d8e3d3c682-kube-api-access-plwpp" pod "304bec98-6219-49cf-bf34-70d8e3d3c682" (UID: "304bec98-6219-49cf-bf34-70d8e3d3c682"). InnerVolumeSpecName "kube-api-access-plwpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:16.517762 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.517741 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bf80c-5dc5c7cfc9-cfdqp"] Nov 03 23:10:16.616202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.616120 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304bec98-6219-49cf-bf34-70d8e3d3c682-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:16.616202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:16.616152 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwpp\" (UniqueName: \"kubernetes.io/projected/304bec98-6219-49cf-bf34-70d8e3d3c682-kube-api-access-plwpp\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.244236 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.244214 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:10:17.321208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.321175 5272 generic.go:356] "Generic (PLEG): container finished" podID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerID="0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407" exitCode=0 Nov 03 23:10:17.321653 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.321231 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" Nov 03 23:10:17.321653 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.321260 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" event={"ID":"10f3ea73-2aeb-40ed-990a-31a494afd1ed","Type":"ContainerDied","Data":"0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407"} Nov 03 23:10:17.321653 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.321321 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8" event={"ID":"10f3ea73-2aeb-40ed-990a-31a494afd1ed","Type":"ContainerDied","Data":"5be775a3e5871f50b499455840a28001db92ba317f2836eda75c59524d717826"} Nov 03 23:10:17.321653 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.321356 5272 scope.go:122] "RemoveContainer" containerID="0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407" Nov 03 23:10:17.322551 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322531 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" Nov 03 23:10:17.322657 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322555 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl" event={"ID":"304bec98-6219-49cf-bf34-70d8e3d3c682","Type":"ContainerDied","Data":"5618f8bf618a43e4083969fa88382a6421de11f7b582de9fa71a7413fa4f7e1f"} Nov 03 23:10:17.322657 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322594 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5618f8bf618a43e4083969fa88382a6421de11f7b582de9fa71a7413fa4f7e1f" Nov 03 23:10:17.322657 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322635 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-credential-socket\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.322884 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322746 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-podinfo\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.322884 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322815 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-data\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.322884 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322875 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-token\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-token\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.323067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322904 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-kube-api-access-s7cwh\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-kube-api-access-s7cwh\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.323067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322937 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-credential-socket" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.323067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.322959 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-socket\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.323224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323201 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-data" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.323224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323210 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-socket" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.323518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323499 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istiod-ca-cert\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.323635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323553 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-certs\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.323635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323585 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-envoy\") pod \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\" (UID: \"10f3ea73-2aeb-40ed-990a-31a494afd1ed\") " Nov 03 23:10:17.323952 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323925 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istiod-ca-cert" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:17.324043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323935 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.324120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.324055 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-certs" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.324120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.323977 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.324120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.324094 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.325293 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.325272 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-token" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:17.325746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.325721 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-podinfo" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:10:17.326038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.326010 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-kube-api-access-s7cwh" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "kube-api-access-s7cwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:17.326273 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.326252 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-envoy" pod "10f3ea73-2aeb-40ed-990a-31a494afd1ed" (UID: "10f3ea73-2aeb-40ed-990a-31a494afd1ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:17.332852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.332830 5272 scope.go:122] "RemoveContainer" containerID="0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972" Nov 03 23:10:17.341517 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.341499 5272 scope.go:122] "RemoveContainer" containerID="0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407" Nov 03 23:10:17.341771 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:10:17.341749 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407\": container with ID starting with 0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407 not found: ID does not exist" containerID="0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407" Nov 03 23:10:17.341838 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.341780 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407"} err="failed to get container status \"0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407\": rpc error: code = NotFound desc = could not find container \"0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407\": container with ID starting with 0fe362747880ed6bd1b4d4077711182d8c68fd78bf27b1b052ed22448740c407 not found: ID does not exist" Nov 03 23:10:17.341838 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.341824 5272 scope.go:122] "RemoveContainer" containerID="0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972" Nov 03 23:10:17.342068 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:10:17.342051 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972\": container with ID starting with 0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972 not found: ID does not exist" containerID="0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972" Nov 03 23:10:17.342112 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.342075 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972"} err="failed to get container status \"0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972\": rpc error: code = NotFound desc = could not find container \"0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972\": container with ID starting with 0c40d0adf57d2f0ac0eb72dcbc94b4ec3cc0232c72c45083e3c0a056fbd28972 not found: ID does not exist" Nov 03 23:10:17.425405 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.425368 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.425405 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.425395 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.425405 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.425405 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7cwh\" (UniqueName: \"kubernetes.io/projected/10f3ea73-2aeb-40ed-990a-31a494afd1ed-kube-api-access-s7cwh\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.425737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.425418 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.425737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.425435 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.425737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.425449 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f3ea73-2aeb-40ed-990a-31a494afd1ed-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:17.492102 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.492064 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" path="/var/lib/kubelet/pods/2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5/volumes" Nov 03 23:10:17.730542 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.730457 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8"] Nov 03 23:10:17.734888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:17.734862 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0bf80c-openshift-default-6566586789-wsnl8"] Nov 03 23:10:19.492294 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:19.492257 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" path="/var/lib/kubelet/pods/10f3ea73-2aeb-40ed-990a-31a494afd1ed/volumes" Nov 03 23:10:24.373819 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:24.373717 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd"] Nov 03 23:10:24.478304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:24.478275 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g"] Nov 03 23:10:24.502957 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:10:24.502919 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:10:29.374777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.374728 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" podUID="942b4c66-7202-4e1a-ae7f-dba0b94de655" containerName="payload-processing" containerID="cri-o://9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27" gracePeriod=25 Nov 03 23:10:29.479090 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.479009 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" podUID="0851de15-f9b3-483e-995c-3af3b18c5310" containerName="payload-pre-processing" containerID="cri-o://2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee" gracePeriod=25 Nov 03 23:10:29.627890 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.627825 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:10:29.753321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.753282 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942b4c66-7202-4e1a-ae7f-dba0b94de655-kube-api-access-56ntz\" (UniqueName: \"kubernetes.io/projected/942b4c66-7202-4e1a-ae7f-dba0b94de655-kube-api-access-56ntz\") pod \"942b4c66-7202-4e1a-ae7f-dba0b94de655\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " Nov 03 23:10:29.753321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.753329 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942b4c66-7202-4e1a-ae7f-dba0b94de655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b4c66-7202-4e1a-ae7f-dba0b94de655-plugins-config-volume\") pod \"942b4c66-7202-4e1a-ae7f-dba0b94de655\" (UID: \"942b4c66-7202-4e1a-ae7f-dba0b94de655\") " Nov 03 23:10:29.753763 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.753737 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942b4c66-7202-4e1a-ae7f-dba0b94de655-plugins-config-volume" pod "942b4c66-7202-4e1a-ae7f-dba0b94de655" (UID: "942b4c66-7202-4e1a-ae7f-dba0b94de655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:29.755438 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.755410 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942b4c66-7202-4e1a-ae7f-dba0b94de655-kube-api-access-56ntz" pod "942b4c66-7202-4e1a-ae7f-dba0b94de655" (UID: "942b4c66-7202-4e1a-ae7f-dba0b94de655"). InnerVolumeSpecName "kube-api-access-56ntz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:29.855006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.854960 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ntz\" (UniqueName: \"kubernetes.io/projected/942b4c66-7202-4e1a-ae7f-dba0b94de655-kube-api-access-56ntz\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:29.855006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:29.854998 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b4c66-7202-4e1a-ae7f-dba0b94de655-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:30.408313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.408272 5272 generic.go:356] "Generic (PLEG): container finished" podID="942b4c66-7202-4e1a-ae7f-dba0b94de655" containerID="9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27" exitCode=0 Nov 03 23:10:30.408739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.408320 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" Nov 03 23:10:30.408739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.408362 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" event={"ID":"942b4c66-7202-4e1a-ae7f-dba0b94de655","Type":"ContainerDied","Data":"9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27"} Nov 03 23:10:30.408739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.408421 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd" event={"ID":"942b4c66-7202-4e1a-ae7f-dba0b94de655","Type":"ContainerDied","Data":"0cb170e0b15717773c451f6daa9ac83c1d614bd9aec76a013814983f490e4b81"} Nov 03 23:10:30.408739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.408444 5272 scope.go:122] "RemoveContainer" containerID="9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27" Nov 03 23:10:30.417833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.417814 5272 scope.go:122] "RemoveContainer" containerID="9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27" Nov 03 23:10:30.418116 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:10:30.418093 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27\": container with ID starting with 9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27 not found: ID does not exist" containerID="9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27" Nov 03 23:10:30.418174 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.418123 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27"} err="failed to get container status \"9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27\": rpc error: code = NotFound desc = could not find container \"9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27\": container with ID starting with 9649c194f57f9c274146f29ce9cf88da96488c270e5e83467015c3ccb6514a27 not found: ID does not exist" Nov 03 23:10:30.561988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.561949 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd"] Nov 03 23:10:30.608537 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:30.608498 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac6633-64d9fbd554-vtdtd"] Nov 03 23:10:31.491008 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:31.490976 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942b4c66-7202-4e1a-ae7f-dba0b94de655" path="/var/lib/kubelet/pods/942b4c66-7202-4e1a-ae7f-dba0b94de655/volumes" Nov 03 23:10:32.308079 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:32.308038 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" podUID="0851de15-f9b3-483e-995c-3af3b18c5310" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:10:37.678350 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:37.678310 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl"] Nov 03 23:10:37.724795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:37.724757 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac6633-qcjnl"] Nov 03 23:10:39.491293 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:39.491249 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304bec98-6219-49cf-bf34-70d8e3d3c682" path="/var/lib/kubelet/pods/304bec98-6219-49cf-bf34-70d8e3d3c682/volumes" Nov 03 23:10:41.049989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:41.049955 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:10:42.307442 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:42.307403 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" podUID="0851de15-f9b3-483e-995c-3af3b18c5310" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:10:45.333682 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.333642 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2"] Nov 03 23:10:45.334122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334061 5272 cpu_manager.go:414] "containerMap: removing container" podUID="942b4c66-7202-4e1a-ae7f-dba0b94de655" containerName="payload-processing" Nov 03 23:10:45.334122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334076 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942b4c66-7202-4e1a-ae7f-dba0b94de655" containerName="payload-processing" Nov 03 23:10:45.334122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334088 5272 cpu_manager.go:414] "containerMap: removing container" podUID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" containerName="payload-pre-processing" Nov 03 23:10:45.334122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334093 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" containerName="payload-pre-processing" Nov 03 23:10:45.334122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334106 5272 cpu_manager.go:414] "containerMap: removing container" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" Nov 03 23:10:45.334122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334111 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" Nov 03 23:10:45.334122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334125 5272 cpu_manager.go:414] "containerMap: removing container" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" Nov 03 23:10:45.334364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334130 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" Nov 03 23:10:45.334364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334147 5272 cpu_manager.go:414] "containerMap: removing container" podUID="304bec98-6219-49cf-bf34-70d8e3d3c682" containerName="revoke-keys" Nov 03 23:10:45.334364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334152 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304bec98-6219-49cf-bf34-70d8e3d3c682" containerName="revoke-keys" Nov 03 23:10:45.334364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334206 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="304bec98-6219-49cf-bf34-70d8e3d3c682" containerName="revoke-keys" Nov 03 23:10:45.334364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334217 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="942b4c66-7202-4e1a-ae7f-dba0b94de655" containerName="payload-processing" Nov 03 23:10:45.334364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334224 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2f344f-859a-4ae4-9dd3-8c6c88ed6aa5" containerName="payload-pre-processing" Nov 03 23:10:45.334364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.334232 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" Nov 03 23:10:45.340291 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.340252 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.391712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.391671 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2"] Nov 03 23:10:45.436005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.435951 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:45.436005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.435972 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:10:45.436249 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.436066 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:10:45.436249 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.436104 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:10:45.506516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.506473 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.506707 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.506663 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mn6v\" (UniqueName: \"kubernetes.io/projected/7dd98ad8-5231-4566-8b7f-e53b14548b9a-kube-api-access-5mn6v\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.506707 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.506695 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.608191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.608100 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mn6v\" (UniqueName: \"kubernetes.io/projected/7dd98ad8-5231-4566-8b7f-e53b14548b9a-kube-api-access-5mn6v\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.608191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.608137 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.608191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.608174 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.608928 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.608909 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.610590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.610558 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.665447 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.665417 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mn6v\" (UniqueName: \"kubernetes.io/projected/7dd98ad8-5231-4566-8b7f-e53b14548b9a-kube-api-access-5mn6v\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:45.956617 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:45.956528 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:46.327384 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:46.327348 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2"] Nov 03 23:10:46.334691 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:10:46.334657 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd98ad8_5231_4566_8b7f_e53b14548b9a.slice/crio-f230d1c8dbde34bc4a5f49ef1e474657240b5a2ccfea5b88b709663d3d9f3701 WatchSource:0}: Error finding container f230d1c8dbde34bc4a5f49ef1e474657240b5a2ccfea5b88b709663d3d9f3701: Status 404 returned error can't find the container with id f230d1c8dbde34bc4a5f49ef1e474657240b5a2ccfea5b88b709663d3d9f3701 Nov 03 23:10:46.491540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:46.491496 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" event={"ID":"7dd98ad8-5231-4566-8b7f-e53b14548b9a","Type":"ContainerStarted","Data":"f230d1c8dbde34bc4a5f49ef1e474657240b5a2ccfea5b88b709663d3d9f3701"} Nov 03 23:10:47.497169 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:47.497083 5272 generic.go:356] "Generic (PLEG): container finished" podID="7dd98ad8-5231-4566-8b7f-e53b14548b9a" containerID="c54d7c84e8e20be765a820e9cd3f195bf9dd3f4b5e77a41f58cb3474b49b1fec" exitCode=0 Nov 03 23:10:47.497537 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:47.497174 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" event={"ID":"7dd98ad8-5231-4566-8b7f-e53b14548b9a","Type":"ContainerDied","Data":"c54d7c84e8e20be765a820e9cd3f195bf9dd3f4b5e77a41f58cb3474b49b1fec"} Nov 03 23:10:48.632476 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.632450 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:48.738506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.738467 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-xks-ca\") pod \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " Nov 03 23:10:48.738714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.738522 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-openshift-ca\") pod \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " Nov 03 23:10:48.738714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.738544 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd98ad8-5231-4566-8b7f-e53b14548b9a-kube-api-access-5mn6v\" (UniqueName: \"kubernetes.io/projected/7dd98ad8-5231-4566-8b7f-e53b14548b9a-kube-api-access-5mn6v\") pod \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\" (UID: \"7dd98ad8-5231-4566-8b7f-e53b14548b9a\") " Nov 03 23:10:48.738990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.738966 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-openshift-ca" pod "7dd98ad8-5231-4566-8b7f-e53b14548b9a" (UID: "7dd98ad8-5231-4566-8b7f-e53b14548b9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:48.740727 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.740701 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd98ad8-5231-4566-8b7f-e53b14548b9a-kube-api-access-5mn6v" pod "7dd98ad8-5231-4566-8b7f-e53b14548b9a" (UID: "7dd98ad8-5231-4566-8b7f-e53b14548b9a"). InnerVolumeSpecName "kube-api-access-5mn6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:48.740871 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.740722 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-xks-ca" pod "7dd98ad8-5231-4566-8b7f-e53b14548b9a" (UID: "7dd98ad8-5231-4566-8b7f-e53b14548b9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:48.839218 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.839181 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:48.839218 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.839211 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dd98ad8-5231-4566-8b7f-e53b14548b9a-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:48.839218 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:48.839225 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mn6v\" (UniqueName: \"kubernetes.io/projected/7dd98ad8-5231-4566-8b7f-e53b14548b9a-kube-api-access-5mn6v\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:49.509283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.509252 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" Nov 03 23:10:49.509283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.509274 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2" event={"ID":"7dd98ad8-5231-4566-8b7f-e53b14548b9a","Type":"ContainerDied","Data":"f230d1c8dbde34bc4a5f49ef1e474657240b5a2ccfea5b88b709663d3d9f3701"} Nov 03 23:10:49.509493 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.509314 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f230d1c8dbde34bc4a5f49ef1e474657240b5a2ccfea5b88b709663d3d9f3701" Nov 03 23:10:49.728784 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.728757 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:10:49.848742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.848709 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0851de15-f9b3-483e-995c-3af3b18c5310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0851de15-f9b3-483e-995c-3af3b18c5310-plugins-config-volume\") pod \"0851de15-f9b3-483e-995c-3af3b18c5310\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " Nov 03 23:10:49.848946 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.848775 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0851de15-f9b3-483e-995c-3af3b18c5310-kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/0851de15-f9b3-483e-995c-3af3b18c5310-kube-api-access-2z7dr\") pod \"0851de15-f9b3-483e-995c-3af3b18c5310\" (UID: \"0851de15-f9b3-483e-995c-3af3b18c5310\") " Nov 03 23:10:49.849181 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.849148 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0851de15-f9b3-483e-995c-3af3b18c5310-plugins-config-volume" pod "0851de15-f9b3-483e-995c-3af3b18c5310" (UID: "0851de15-f9b3-483e-995c-3af3b18c5310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:49.851424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.851388 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0851de15-f9b3-483e-995c-3af3b18c5310-kube-api-access-2z7dr" pod "0851de15-f9b3-483e-995c-3af3b18c5310" (UID: "0851de15-f9b3-483e-995c-3af3b18c5310"). InnerVolumeSpecName "kube-api-access-2z7dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:49.949567 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.949527 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/0851de15-f9b3-483e-995c-3af3b18c5310-kube-api-access-2z7dr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:49.949567 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:49.949559 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0851de15-f9b3-483e-995c-3af3b18c5310-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:50.516049 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.516008 5272 generic.go:356] "Generic (PLEG): container finished" podID="0851de15-f9b3-483e-995c-3af3b18c5310" containerID="2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee" exitCode=0 Nov 03 23:10:50.516235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.516073 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" Nov 03 23:10:50.516235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.516087 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" event={"ID":"0851de15-f9b3-483e-995c-3af3b18c5310","Type":"ContainerDied","Data":"2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee"} Nov 03 23:10:50.516235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.516140 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g" event={"ID":"0851de15-f9b3-483e-995c-3af3b18c5310","Type":"ContainerDied","Data":"8b022e9c47744b6ef14f1dd1ca3e26b2e7d131b09f3e0ebe3fdab7894615ccf7"} Nov 03 23:10:50.516235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.516161 5272 scope.go:122] "RemoveContainer" containerID="2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee" Nov 03 23:10:50.526161 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.526140 5272 scope.go:122] "RemoveContainer" containerID="2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee" Nov 03 23:10:50.526460 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:10:50.526437 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee\": container with ID starting with 2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee not found: ID does not exist" containerID="2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee" Nov 03 23:10:50.526516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.526470 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee"} err="failed to get container status \"2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee\": rpc error: code = NotFound desc = could not find container \"2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee\": container with ID starting with 2d6c37ba692bd58cb7969667a4544b3779b253763b04b670dc1b151d420896ee not found: ID does not exist" Nov 03 23:10:50.669411 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.669378 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g"] Nov 03 23:10:50.715679 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:50.715641 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac6633-7bf6856fbc-5nf4g"] Nov 03 23:10:51.490908 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:51.490867 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0851de15-f9b3-483e-995c-3af3b18c5310" path="/var/lib/kubelet/pods/0851de15-f9b3-483e-995c-3af3b18c5310/volumes" Nov 03 23:10:54.500207 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:10:54.500168 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:10:56.873019 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.872974 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7"] Nov 03 23:10:56.882370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882336 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf"] Nov 03 23:10:56.882715 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882702 5272 cpu_manager.go:414] "containerMap: removing container" podUID="7dd98ad8-5231-4566-8b7f-e53b14548b9a" containerName="revoke-keys" Nov 03 23:10:56.882761 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882716 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd98ad8-5231-4566-8b7f-e53b14548b9a" containerName="revoke-keys" Nov 03 23:10:56.882761 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882745 5272 cpu_manager.go:414] "containerMap: removing container" podUID="0851de15-f9b3-483e-995c-3af3b18c5310" containerName="payload-pre-processing" Nov 03 23:10:56.882761 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882750 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0851de15-f9b3-483e-995c-3af3b18c5310" containerName="payload-pre-processing" Nov 03 23:10:56.882887 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882817 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd98ad8-5231-4566-8b7f-e53b14548b9a" containerName="revoke-keys" Nov 03 23:10:56.882887 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882829 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f3ea73-2aeb-40ed-990a-31a494afd1ed" containerName="istio-proxy" Nov 03 23:10:56.882887 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.882839 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="0851de15-f9b3-483e-995c-3af3b18c5310" containerName="payload-pre-processing" Nov 03 23:10:56.886975 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.886954 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:56.915909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.915875 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gstg\" (UniqueName: \"kubernetes.io/projected/723d2056-0ebe-4a91-9168-dc7d58211f48-kube-api-access-2gstg\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:56.916052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.915938 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:56.916052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.916034 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:56.937812 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.937783 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:10:56.940195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.940175 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:56.970184 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.970156 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:10:56.970311 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.970206 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:10:56.979254 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:56.979224 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf"] Nov 03 23:10:57.016880 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.016844 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gstg\" (UniqueName: \"kubernetes.io/projected/723d2056-0ebe-4a91-9168-dc7d58211f48-kube-api-access-2gstg\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:57.017052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.016896 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:57.017052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.016933 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:57.017515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.017496 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:57.019439 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.019416 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:57.045920 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.045892 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s"] Nov 03 23:10:57.048118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.048097 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28692ad6-67x9s"] Nov 03 23:10:57.074388 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.074355 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gstg\" (UniqueName: \"kubernetes.io/projected/723d2056-0ebe-4a91-9168-dc7d58211f48-kube-api-access-2gstg\") pod \"maas-api-revoke-sub-3cf66932-8pnsf\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:57.199486 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.199399 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:57.492205 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.492127 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="627d1908-fcf8-4d9d-bb70-22fbf83f8d7e" path="/var/lib/kubelet/pods/627d1908-fcf8-4d9d-bb70-22fbf83f8d7e/volumes" Nov 03 23:10:57.574782 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:57.574750 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf"] Nov 03 23:10:57.582470 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:10:57.582440 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723d2056_0ebe_4a91_9168_dc7d58211f48.slice/crio-3a6bd2b8c8be6c97dc1cfa18fe67015d6f4a0fb9e99ed777098e5283e1304f0f WatchSource:0}: Error finding container 3a6bd2b8c8be6c97dc1cfa18fe67015d6f4a0fb9e99ed777098e5283e1304f0f: Status 404 returned error can't find the container with id 3a6bd2b8c8be6c97dc1cfa18fe67015d6f4a0fb9e99ed777098e5283e1304f0f Nov 03 23:10:58.561025 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:58.560992 5272 generic.go:356] "Generic (PLEG): container finished" podID="723d2056-0ebe-4a91-9168-dc7d58211f48" containerID="3f83d994538003121240d2908c4286c508f851fe31a6cf20a4ae80c85524b520" exitCode=0 Nov 03 23:10:58.561422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:58.561065 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" event={"ID":"723d2056-0ebe-4a91-9168-dc7d58211f48","Type":"ContainerDied","Data":"3f83d994538003121240d2908c4286c508f851fe31a6cf20a4ae80c85524b520"} Nov 03 23:10:58.561422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:58.561102 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" event={"ID":"723d2056-0ebe-4a91-9168-dc7d58211f48","Type":"ContainerStarted","Data":"3a6bd2b8c8be6c97dc1cfa18fe67015d6f4a0fb9e99ed777098e5283e1304f0f"} Nov 03 23:10:59.699778 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.699742 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:10:59.745898 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.745860 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-xks-ca\") pod \"723d2056-0ebe-4a91-9168-dc7d58211f48\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " Nov 03 23:10:59.746098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.745932 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-openshift-ca\") pod \"723d2056-0ebe-4a91-9168-dc7d58211f48\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " Nov 03 23:10:59.746098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.745987 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723d2056-0ebe-4a91-9168-dc7d58211f48-kube-api-access-2gstg\" (UniqueName: \"kubernetes.io/projected/723d2056-0ebe-4a91-9168-dc7d58211f48-kube-api-access-2gstg\") pod \"723d2056-0ebe-4a91-9168-dc7d58211f48\" (UID: \"723d2056-0ebe-4a91-9168-dc7d58211f48\") " Nov 03 23:10:59.746365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.746333 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-openshift-ca" pod "723d2056-0ebe-4a91-9168-dc7d58211f48" (UID: "723d2056-0ebe-4a91-9168-dc7d58211f48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:59.748039 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.748010 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723d2056-0ebe-4a91-9168-dc7d58211f48-kube-api-access-2gstg" pod "723d2056-0ebe-4a91-9168-dc7d58211f48" (UID: "723d2056-0ebe-4a91-9168-dc7d58211f48"). InnerVolumeSpecName "kube-api-access-2gstg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:59.748039 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.748024 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-xks-ca" pod "723d2056-0ebe-4a91-9168-dc7d58211f48" (UID: "723d2056-0ebe-4a91-9168-dc7d58211f48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:59.847623 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.847537 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gstg\" (UniqueName: \"kubernetes.io/projected/723d2056-0ebe-4a91-9168-dc7d58211f48-kube-api-access-2gstg\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:59.847623 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.847568 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:59.847623 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:10:59.847579 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/723d2056-0ebe-4a91-9168-dc7d58211f48-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:00.573728 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:00.573693 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" Nov 03 23:11:00.573943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:00.573732 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf" event={"ID":"723d2056-0ebe-4a91-9168-dc7d58211f48","Type":"ContainerDied","Data":"3a6bd2b8c8be6c97dc1cfa18fe67015d6f4a0fb9e99ed777098e5283e1304f0f"} Nov 03 23:11:00.573943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:00.573789 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6bd2b8c8be6c97dc1cfa18fe67015d6f4a0fb9e99ed777098e5283e1304f0f" Nov 03 23:11:01.873415 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:01.873364 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" podUID="db0e8e22-ee36-4264-89f3-36c93ac79c62" containerName="payload-processing" containerID="cri-o://6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2" gracePeriod=25 Nov 03 23:11:09.832618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:09.832580 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" podUID="db0e8e22-ee36-4264-89f3-36c93ac79c62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:11:10.047654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:10.047609 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8"] Nov 03 23:11:10.094286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:10.094205 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e05e65-gdrq8"] Nov 03 23:11:11.491444 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:11.491408 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf38b79-e49e-4974-a43e-809ea6bed7cf" path="/var/lib/kubelet/pods/ecf38b79-e49e-4974-a43e-809ea6bed7cf/volumes" Nov 03 23:11:13.750210 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:13.750170 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:11:19.832168 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:19.832128 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" podUID="db0e8e22-ee36-4264-89f3-36c93ac79c62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:11:22.123786 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.123759 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 23:11:22.262118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.262040 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0e8e22-ee36-4264-89f3-36c93ac79c62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0e8e22-ee36-4264-89f3-36c93ac79c62-plugins-config-volume\") pod \"db0e8e22-ee36-4264-89f3-36c93ac79c62\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " Nov 03 23:11:22.262298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.262130 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0e8e22-ee36-4264-89f3-36c93ac79c62-kube-api-access-mkbwl\" (UniqueName: \"kubernetes.io/projected/db0e8e22-ee36-4264-89f3-36c93ac79c62-kube-api-access-mkbwl\") pod \"db0e8e22-ee36-4264-89f3-36c93ac79c62\" (UID: \"db0e8e22-ee36-4264-89f3-36c93ac79c62\") " Nov 03 23:11:22.262384 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.262361 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0e8e22-ee36-4264-89f3-36c93ac79c62-plugins-config-volume" pod "db0e8e22-ee36-4264-89f3-36c93ac79c62" (UID: "db0e8e22-ee36-4264-89f3-36c93ac79c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:22.264262 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.264229 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0e8e22-ee36-4264-89f3-36c93ac79c62-kube-api-access-mkbwl" pod "db0e8e22-ee36-4264-89f3-36c93ac79c62" (UID: "db0e8e22-ee36-4264-89f3-36c93ac79c62"). InnerVolumeSpecName "kube-api-access-mkbwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:22.362981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.362940 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkbwl\" (UniqueName: \"kubernetes.io/projected/db0e8e22-ee36-4264-89f3-36c93ac79c62-kube-api-access-mkbwl\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:22.362981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.362973 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0e8e22-ee36-4264-89f3-36c93ac79c62-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:22.690341 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.690310 5272 generic.go:356] "Generic (PLEG): container finished" podID="db0e8e22-ee36-4264-89f3-36c93ac79c62" containerID="6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2" exitCode=0 Nov 03 23:11:22.690570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.690355 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" Nov 03 23:11:22.690570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.690400 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" event={"ID":"db0e8e22-ee36-4264-89f3-36c93ac79c62","Type":"ContainerDied","Data":"6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2"} Nov 03 23:11:22.690570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.690457 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7" event={"ID":"db0e8e22-ee36-4264-89f3-36c93ac79c62","Type":"ContainerDied","Data":"f562e3710b962ff06e8425a8edfb2cd072090cde8eb723610656145c6ec9f7f1"} Nov 03 23:11:22.690570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.690487 5272 scope.go:122] "RemoveContainer" containerID="6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2" Nov 03 23:11:22.699620 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.699597 5272 scope.go:122] "RemoveContainer" containerID="6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2" Nov 03 23:11:22.699911 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:11:22.699892 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2\": container with ID starting with 6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2 not found: ID does not exist" containerID="6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2" Nov 03 23:11:22.699978 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.699921 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2"} err="failed to get container status \"6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2\": rpc error: code = NotFound desc = could not find container \"6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2\": container with ID starting with 6a47a36cfd651fcfd612aa901dfab0bccae810c6a52c5ee31d018abcf99172e2 not found: ID does not exist" Nov 03 23:11:22.843899 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.843863 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7"] Nov 03 23:11:22.890432 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:22.890397 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f1190-c8f5948fd-sd9d7"] Nov 03 23:11:23.491155 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:23.491122 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0e8e22-ee36-4264-89f3-36c93ac79c62" path="/var/lib/kubelet/pods/db0e8e22-ee36-4264-89f3-36c93ac79c62/volumes" Nov 03 23:11:24.500281 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:11:24.500241 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:11:31.839177 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:31.839138 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:11:44.049618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:44.049582 5272 scope.go:122] "RemoveContainer" containerID="cf6f34144c9dd9e9b779c0321dca552334a823c17d8156688eff3db0a8ec133a" Nov 03 23:11:44.050357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:44.050334 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2"] Nov 03 23:11:44.096918 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:44.096887 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f1190-qtjf2"] Nov 03 23:11:45.491344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:45.491310 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd98ad8-5231-4566-8b7f-e53b14548b9a" path="/var/lib/kubelet/pods/7dd98ad8-5231-4566-8b7f-e53b14548b9a/volumes" Nov 03 23:11:47.284896 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:47.284859 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:11:50.703196 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:50.703118 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz"] Nov 03 23:11:50.703668 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:50.703334 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" containerID="cri-o://cff628ebda889502123c28fe1b59495d042053682283895b6cdc2572987ee4a0" gracePeriod=30 Nov 03 23:11:51.286842 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:51.286792 5272 patch_prober.go:29] interesting pod/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:11:51.287019 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:51.286876 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:11:53.044950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:53.044909 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv"] Nov 03 23:11:53.094232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:53.094177 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv"] Nov 03 23:11:53.100418 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:53.100372 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c96d85-w6lrv"] Nov 03 23:11:53.101583 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:53.101559 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ef1667-vt6nv"] Nov 03 23:11:53.491746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:53.491668 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1b5305-3e01-43cd-aad8-25756270a140" path="/var/lib/kubelet/pods/bb1b5305-3e01-43cd-aad8-25756270a140/volumes" Nov 03 23:11:53.492116 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:53.492100 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3" path="/var/lib/kubelet/pods/fba6c351-f8c4-4f9c-9f94-4b8fc736d7f3/volumes" Nov 03 23:11:54.500833 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:11:54.500773 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:11:55.879038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:55.879009 5272 generic.go:356] "Generic (PLEG): container finished" podID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerID="cff628ebda889502123c28fe1b59495d042053682283895b6cdc2572987ee4a0" exitCode=0 Nov 03 23:11:55.879413 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:55.879077 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" event={"ID":"d875c132-fdfc-4f22-8167-cbbb8c44fffe","Type":"ContainerDied","Data":"cff628ebda889502123c28fe1b59495d042053682283895b6cdc2572987ee4a0"} Nov 03 23:11:55.879413 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:55.879149 5272 scope.go:122] "RemoveContainer" containerID="89be454ba77027205e2e7c89dc7c055e03a11c5458dbba424d35c71e3ba28238" Nov 03 23:11:55.946531 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:55.946503 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 23:11:56.069347 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069317 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-kube-api-access-dwn7v\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-kube-api-access-dwn7v\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069528 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069361 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-certs\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069528 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069401 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-credential-socket\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069528 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069464 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-envoy\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069528 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069507 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-token\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-token\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069590 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-socket\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069620 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-data\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069661 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istiod-ca-cert\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069691 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-podinfo\") pod \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\" (UID: \"d875c132-fdfc-4f22-8167-cbbb8c44fffe\") " Nov 03 23:11:56.069760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069709 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-certs" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:56.070059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069773 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-credential-socket" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:56.070059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.069996 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-data" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:56.070164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.070067 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.070164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.070090 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.070243 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.070166 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-socket" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:56.070243 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.070084 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istiod-ca-cert" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:56.071773 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.071749 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-envoy" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:11:56.071905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.071789 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-podinfo" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:11:56.071905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.071825 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-kube-api-access-dwn7v" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "kube-api-access-dwn7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:56.071905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.071853 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-token" pod "d875c132-fdfc-4f22-8167-cbbb8c44fffe" (UID: "d875c132-fdfc-4f22-8167-cbbb8c44fffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:56.171235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.171190 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.171235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.171228 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.171235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.171238 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.171496 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.171247 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.171496 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.171257 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.171496 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.171265 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d875c132-fdfc-4f22-8167-cbbb8c44fffe-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.171496 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.171274 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwn7v\" (UniqueName: \"kubernetes.io/projected/d875c132-fdfc-4f22-8167-cbbb8c44fffe-kube-api-access-dwn7v\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:11:56.886037 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.885998 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" event={"ID":"d875c132-fdfc-4f22-8167-cbbb8c44fffe","Type":"ContainerDied","Data":"abd446ab5cb1d6c00535b75e7080fc8252d72a5135f7033984ae83cda3725ab6"} Nov 03 23:11:56.886037 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.886028 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz" Nov 03 23:11:56.886558 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:56.886060 5272 scope.go:122] "RemoveContainer" containerID="cff628ebda889502123c28fe1b59495d042053682283895b6cdc2572987ee4a0" Nov 03 23:11:57.040592 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:57.040555 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz"] Nov 03 23:11:57.087005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:57.086972 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f1190-openshift-default-59fcbbc8d5-mzhsz"] Nov 03 23:11:57.490968 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:11:57.490926 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" path="/var/lib/kubelet/pods/d875c132-fdfc-4f22-8167-cbbb8c44fffe/volumes" Nov 03 23:12:24.500743 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:12:24.500696 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:12:29.320200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320158 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz"] Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320518 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320530 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320542 5272 cpu_manager.go:414] "containerMap: removing container" podUID="723d2056-0ebe-4a91-9168-dc7d58211f48" containerName="revoke-keys" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320547 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723d2056-0ebe-4a91-9168-dc7d58211f48" containerName="revoke-keys" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320571 5272 cpu_manager.go:414] "containerMap: removing container" podUID="db0e8e22-ee36-4264-89f3-36c93ac79c62" containerName="payload-processing" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320577 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0e8e22-ee36-4264-89f3-36c93ac79c62" containerName="payload-processing" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320640 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320648 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="723d2056-0ebe-4a91-9168-dc7d58211f48" containerName="revoke-keys" Nov 03 23:12:29.320698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.320659 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0e8e22-ee36-4264-89f3-36c93ac79c62" containerName="payload-processing" Nov 03 23:12:29.323417 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.323395 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.372366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.372312 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:12:29.391783 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.391753 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.391931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.391813 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.391985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.391919 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4cm\" (UniqueName: \"kubernetes.io/projected/4a1558a0-57b5-47f0-b1c4-87ca2246a307-kube-api-access-md4cm\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.418955 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.418926 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:12:29.418955 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.418935 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:12:29.419124 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.419084 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:12:29.420532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.420511 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz"] Nov 03 23:12:29.492590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.492560 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4cm\" (UniqueName: \"kubernetes.io/projected/4a1558a0-57b5-47f0-b1c4-87ca2246a307-kube-api-access-md4cm\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.492758 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.492656 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.492758 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.492687 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.493398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.493379 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.495118 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.495095 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.550016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.549984 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4cm\" (UniqueName: \"kubernetes.io/projected/4a1558a0-57b5-47f0-b1c4-87ca2246a307-kube-api-access-md4cm\") pod \"maas-api-revoke-sub-85237e45-fnpfz\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:29.636079 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:29.635997 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:30.007262 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:30.007178 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz"] Nov 03 23:12:30.014724 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:12:30.014691 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1558a0_57b5_47f0_b1c4_87ca2246a307.slice/crio-867ec526cc985f761723dd76afbc847b7c3136e6d9a468a73a2449d5f506d60d WatchSource:0}: Error finding container 867ec526cc985f761723dd76afbc847b7c3136e6d9a468a73a2449d5f506d60d: Status 404 returned error can't find the container with id 867ec526cc985f761723dd76afbc847b7c3136e6d9a468a73a2449d5f506d60d Nov 03 23:12:30.060660 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:30.060626 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" event={"ID":"4a1558a0-57b5-47f0-b1c4-87ca2246a307","Type":"ContainerStarted","Data":"867ec526cc985f761723dd76afbc847b7c3136e6d9a468a73a2449d5f506d60d"} Nov 03 23:12:31.067370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:31.067280 5272 generic.go:356] "Generic (PLEG): container finished" podID="4a1558a0-57b5-47f0-b1c4-87ca2246a307" containerID="00961b035cab0653308cd999265becc8debd84aecf1dbbaa304412765c0ea759" exitCode=0 Nov 03 23:12:31.067370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:31.067322 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" event={"ID":"4a1558a0-57b5-47f0-b1c4-87ca2246a307","Type":"ContainerDied","Data":"00961b035cab0653308cd999265becc8debd84aecf1dbbaa304412765c0ea759"} Nov 03 23:12:32.202394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.202363 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:32.317795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.317764 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-xks-ca\") pod \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " Nov 03 23:12:32.318017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.317855 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1558a0-57b5-47f0-b1c4-87ca2246a307-kube-api-access-md4cm\" (UniqueName: \"kubernetes.io/projected/4a1558a0-57b5-47f0-b1c4-87ca2246a307-kube-api-access-md4cm\") pod \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " Nov 03 23:12:32.318017 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.317913 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-openshift-ca\") pod \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\" (UID: \"4a1558a0-57b5-47f0-b1c4-87ca2246a307\") " Nov 03 23:12:32.318261 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.318239 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-openshift-ca" pod "4a1558a0-57b5-47f0-b1c4-87ca2246a307" (UID: "4a1558a0-57b5-47f0-b1c4-87ca2246a307"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:12:32.320005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.319974 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1558a0-57b5-47f0-b1c4-87ca2246a307-kube-api-access-md4cm" pod "4a1558a0-57b5-47f0-b1c4-87ca2246a307" (UID: "4a1558a0-57b5-47f0-b1c4-87ca2246a307"). InnerVolumeSpecName "kube-api-access-md4cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:12:32.320005 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.319989 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-xks-ca" pod "4a1558a0-57b5-47f0-b1c4-87ca2246a307" (UID: "4a1558a0-57b5-47f0-b1c4-87ca2246a307"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:32.418908 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.418876 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md4cm\" (UniqueName: \"kubernetes.io/projected/4a1558a0-57b5-47f0-b1c4-87ca2246a307-kube-api-access-md4cm\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:32.418908 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.418904 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:32.418908 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:32.418914 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1558a0-57b5-47f0-b1c4-87ca2246a307-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:12:33.077733 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:33.077692 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" event={"ID":"4a1558a0-57b5-47f0-b1c4-87ca2246a307","Type":"ContainerDied","Data":"867ec526cc985f761723dd76afbc847b7c3136e6d9a468a73a2449d5f506d60d"} Nov 03 23:12:33.077955 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:33.077743 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="867ec526cc985f761723dd76afbc847b7c3136e6d9a468a73a2449d5f506d60d" Nov 03 23:12:33.077955 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:33.077701 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz" Nov 03 23:12:34.387823 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.387769 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22"] Nov 03 23:12:34.388230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.388151 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4a1558a0-57b5-47f0-b1c4-87ca2246a307" containerName="revoke-keys" Nov 03 23:12:34.388230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.388164 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1558a0-57b5-47f0-b1c4-87ca2246a307" containerName="revoke-keys" Nov 03 23:12:34.388230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.388196 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" Nov 03 23:12:34.388230 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.388205 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" Nov 03 23:12:34.388365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.388275 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d875c132-fdfc-4f22-8167-cbbb8c44fffe" containerName="istio-proxy" Nov 03 23:12:34.388365 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.388289 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1558a0-57b5-47f0-b1c4-87ca2246a307" containerName="revoke-keys" Nov 03 23:12:34.391457 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.391435 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.489258 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.489214 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22"] Nov 03 23:12:34.537179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537140 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537186 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537214 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537322 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537487 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537357 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfqp4\" (UniqueName: \"kubernetes.io/projected/cbd938fa-3e39-4e39-a30e-4d2fa083d588-kube-api-access-kfqp4\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537487 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537390 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537487 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537481 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537602 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537503 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.537602 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.537549 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639175 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639093 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639245 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639286 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639308 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639349 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639371 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqp4\" (UniqueName: \"kubernetes.io/projected/cbd938fa-3e39-4e39-a30e-4d2fa083d588-kube-api-access-kfqp4\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639403 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639471 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639504 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639521 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639579 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.639872 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.639849 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.640117 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.640092 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.640200 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.640174 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.641644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.641621 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.642437 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.642415 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.697737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.697703 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd938fa-3e39-4e39-a30e-4d2fa083d588-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.697915 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.697868 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqp4\" (UniqueName: \"kubernetes.io/projected/cbd938fa-3e39-4e39-a30e-4d2fa083d588-kube-api-access-kfqp4\") pod \"maas-default-gateway-openshift-default-59f957fdd7-79h22\" (UID: \"cbd938fa-3e39-4e39-a30e-4d2fa083d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:34.704529 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:34.704499 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:35.130212 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:35.130175 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22"] Nov 03 23:12:35.137944 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:12:35.137915 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd938fa_3e39_4e39_a30e_4d2fa083d588.slice/crio-353d3d9bfb5690e0211a522d1f3d9dfee8b988352b23a7e08b332ba0e55773ae WatchSource:0}: Error finding container 353d3d9bfb5690e0211a522d1f3d9dfee8b988352b23a7e08b332ba0e55773ae: Status 404 returned error can't find the container with id 353d3d9bfb5690e0211a522d1f3d9dfee8b988352b23a7e08b332ba0e55773ae Nov 03 23:12:35.140422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:35.140385 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:35.140495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:35.140459 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:35.140495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:35.140488 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:36.096979 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:36.096939 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" event={"ID":"cbd938fa-3e39-4e39-a30e-4d2fa083d588","Type":"ContainerStarted","Data":"01ccd496031acd25574787fa8816e7a1a6bb585d21c3bfd19043d707b248f949"} Nov 03 23:12:36.097382 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:36.097029 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" event={"ID":"cbd938fa-3e39-4e39-a30e-4d2fa083d588","Type":"ContainerStarted","Data":"353d3d9bfb5690e0211a522d1f3d9dfee8b988352b23a7e08b332ba0e55773ae"} Nov 03 23:12:36.270865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:36.270776 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podStartSLOduration=2.27075688 podStartE2EDuration="2.27075688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:36.269175974 +0000 UTC m=+2936.891484295" watchObservedRunningTime="2026-11-03 23:12:36.27075688 +0000 UTC m=+2936.893065189" Nov 03 23:12:36.705712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:36.705669 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:36.707167 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:36.707139 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-79h22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 03 23:12:36.707312 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:36.707188 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podUID="cbd938fa-3e39-4e39-a30e-4d2fa083d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 03 23:12:37.705788 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:37.705751 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-79h22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 03 23:12:37.706181 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:37.705825 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podUID="cbd938fa-3e39-4e39-a30e-4d2fa083d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 03 23:12:38.704984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:38.704937 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-79h22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 03 23:12:38.705191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:38.705001 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podUID="cbd938fa-3e39-4e39-a30e-4d2fa083d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 03 23:12:39.705247 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:39.705205 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-79h22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 03 23:12:39.705670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:39.705272 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podUID="cbd938fa-3e39-4e39-a30e-4d2fa083d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 03 23:12:40.704856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:40.704791 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-79h22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 03 23:12:40.705054 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:40.704876 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podUID="cbd938fa-3e39-4e39-a30e-4d2fa083d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 03 23:12:41.705691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:41.705646 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-79h22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 03 23:12:41.706141 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:41.705720 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podUID="cbd938fa-3e39-4e39-a30e-4d2fa083d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 03 23:12:43.423038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:43.422998 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:43.423688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:43.423664 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:43.424635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:43.424611 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" Nov 03 23:12:44.079148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:44.079111 5272 scope.go:122] "RemoveContainer" containerID="8c9f87c4335f19bc6896ed38cda51a843f13b7133645107044a704d0abb93cf6" Nov 03 23:12:44.088346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:44.088318 5272 scope.go:122] "RemoveContainer" containerID="c6bca53a44bed1b600386e778ecb3c6b9a9ffb8f753461e301a447e1e5f02888" Nov 03 23:12:44.096573 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:44.096547 5272 scope.go:122] "RemoveContainer" containerID="707f66bcf62de45185eeb8e333967ba08294a3aaeab1abf37d0be53e89dce6fd" Nov 03 23:12:44.104945 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:44.104922 5272 scope.go:122] "RemoveContainer" containerID="41df2883e791308fa227d82031d3e4731a0ccb7e3bfc5efd65a29b1b21321fd5" Nov 03 23:12:46.045614 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:46.045580 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5"] Nov 03 23:12:46.092148 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:46.092113 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9"] Nov 03 23:12:46.093489 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:46.093470 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1a6d2b-p8sx5"] Nov 03 23:12:46.094735 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:46.094712 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e0025c2-dgdd9"] Nov 03 23:12:47.490990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:47.490959 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a9b2f3-51f3-4d14-b548-8a030b96f6b4" path="/var/lib/kubelet/pods/54a9b2f3-51f3-4d14-b548-8a030b96f6b4/volumes" Nov 03 23:12:47.491376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:47.491353 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b46c2c-8ad7-4a93-a377-7e8c2e6f2724" path="/var/lib/kubelet/pods/55b46c2c-8ad7-4a93-a377-7e8c2e6f2724/volumes" Nov 03 23:12:49.514244 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.514210 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft"] Nov 03 23:12:49.518001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.517980 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.590942 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.590910 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppqt\" (UniqueName: \"kubernetes.io/projected/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-kube-api-access-sppqt\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591169 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591075 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591169 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591123 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591234 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591263 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591281 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591456 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591345 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591456 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591418 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.591535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.591467 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.608719 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.608686 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft"] Nov 03 23:12:49.692866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.692827 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.692877 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.692894 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.692911 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.692928 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.692947 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.692971 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppqt\" (UniqueName: \"kubernetes.io/projected/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-kube-api-access-sppqt\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.693030 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.693044 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693552 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.693525 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.693605 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.693652 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693780 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.693754 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.693998 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.693978 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.695577 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.695549 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.695702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.695685 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.751468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.751437 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.751667 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.751644 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppqt\" (UniqueName: \"kubernetes.io/projected/c047281a-ef57-4b5b-9db3-4947c5dbe7aa-kube-api-access-sppqt\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qrmft\" (UID: \"c047281a-ef57-4b5b-9db3-4947c5dbe7aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:49.830618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:49.830585 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:50.257682 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:50.257594 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft"] Nov 03 23:12:50.265028 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:12:50.264999 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc047281a_ef57_4b5b_9db3_4947c5dbe7aa.slice/crio-377c0244af8b581cdf95613c4462cbecadc31094cbe328b249018f7f98cc1076 WatchSource:0}: Error finding container 377c0244af8b581cdf95613c4462cbecadc31094cbe328b249018f7f98cc1076: Status 404 returned error can't find the container with id 377c0244af8b581cdf95613c4462cbecadc31094cbe328b249018f7f98cc1076 Nov 03 23:12:50.267176 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:50.267141 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:50.267265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:50.267211 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:50.267265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:50.267241 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:12:51.177441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:51.177400 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" event={"ID":"c047281a-ef57-4b5b-9db3-4947c5dbe7aa","Type":"ContainerStarted","Data":"83801e90a612b8e8946081a1adc82cde82217149430f0c864b8d925619c4d725"} Nov 03 23:12:51.177945 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:51.177474 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" event={"ID":"c047281a-ef57-4b5b-9db3-4947c5dbe7aa","Type":"ContainerStarted","Data":"377c0244af8b581cdf95613c4462cbecadc31094cbe328b249018f7f98cc1076"} Nov 03 23:12:51.339211 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:51.339153 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podStartSLOduration=2.339138948 podStartE2EDuration="2.339138948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:51.338694304 +0000 UTC m=+2951.961002611" watchObservedRunningTime="2026-11-03 23:12:51.339138948 +0000 UTC m=+2951.961447253" Nov 03 23:12:51.831540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:51.831493 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:51.833124 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:51.833099 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qrmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:12:51.833199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:51.833150 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podUID="c047281a-ef57-4b5b-9db3-4947c5dbe7aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:12:52.831407 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:52.831369 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qrmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:12:52.831836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:52.831427 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podUID="c047281a-ef57-4b5b-9db3-4947c5dbe7aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:12:53.831534 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:53.831488 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qrmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:12:53.831991 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:53.831566 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podUID="c047281a-ef57-4b5b-9db3-4947c5dbe7aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:12:54.500265 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:12:54.500222 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:12:54.831129 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:54.831090 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qrmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:12:54.831338 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:54.831145 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podUID="c047281a-ef57-4b5b-9db3-4947c5dbe7aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:12:55.831869 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:55.831830 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qrmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:12:55.832364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:55.831897 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podUID="c047281a-ef57-4b5b-9db3-4947c5dbe7aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:12:56.831709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:56.831670 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qrmft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Nov 03 23:12:56.831929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:56.831729 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podUID="c047281a-ef57-4b5b-9db3-4947c5dbe7aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Nov 03 23:12:58.519026 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:58.518975 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:58.519541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:58.519257 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:12:58.520337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:12:58.520318 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" Nov 03 23:13:24.499234 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:13:24.499159 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:13:32.570375 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:32.570338 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:13:41.140126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.140088 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z"] Nov 03 23:13:41.143462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.143437 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.177269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.177245 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4nr\" (UniqueName: \"kubernetes.io/projected/16ef3981-0023-4948-9df7-1ee089a9d99e-kube-api-access-mz4nr\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.177394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.177281 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.177394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.177307 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.238861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.238832 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:13:41.238861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.238830 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:13:41.239057 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.238830 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:13:41.239057 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.238861 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:13:41.240501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.240480 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z"] Nov 03 23:13:41.279353 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.279327 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.279491 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.279366 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.279535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.279505 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4nr\" (UniqueName: \"kubernetes.io/projected/16ef3981-0023-4948-9df7-1ee089a9d99e-kube-api-access-mz4nr\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.280075 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.280054 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.281724 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.281696 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.337052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.337024 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4nr\" (UniqueName: \"kubernetes.io/projected/16ef3981-0023-4948-9df7-1ee089a9d99e-kube-api-access-mz4nr\") pod \"maas-api-revoke-sub-c1e271a9-nph6z\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.455850 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.455770 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:41.828611 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:41.828576 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z"] Nov 03 23:13:41.835850 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:13:41.835814 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ef3981_0023_4948_9df7_1ee089a9d99e.slice/crio-6587654bf24167bc51489384b2b3224f621b5c7b9e98fec3b7dd8cdedff6b31d WatchSource:0}: Error finding container 6587654bf24167bc51489384b2b3224f621b5c7b9e98fec3b7dd8cdedff6b31d: Status 404 returned error can't find the container with id 6587654bf24167bc51489384b2b3224f621b5c7b9e98fec3b7dd8cdedff6b31d Nov 03 23:13:42.454296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:42.454211 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" event={"ID":"16ef3981-0023-4948-9df7-1ee089a9d99e","Type":"ContainerStarted","Data":"bca6203b28d6161e2788a99f602e46d1b57c77ccde22800de7bc856fca9ce178"} Nov 03 23:13:42.454296 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:42.454280 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" event={"ID":"16ef3981-0023-4948-9df7-1ee089a9d99e","Type":"ContainerStarted","Data":"6587654bf24167bc51489384b2b3224f621b5c7b9e98fec3b7dd8cdedff6b31d"} Nov 03 23:13:42.563542 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:42.563486 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" podStartSLOduration=1.563466836 podStartE2EDuration="1.563466836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:13:42.562476181 +0000 UTC m=+3003.184784490" watchObservedRunningTime="2026-11-03 23:13:42.563466836 +0000 UTC m=+3003.185775214" Nov 03 23:13:43.460434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:43.460394 5272 generic.go:356] "Generic (PLEG): container finished" podID="16ef3981-0023-4948-9df7-1ee089a9d99e" containerID="bca6203b28d6161e2788a99f602e46d1b57c77ccde22800de7bc856fca9ce178" exitCode=0 Nov 03 23:13:43.460874 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:43.460485 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" event={"ID":"16ef3981-0023-4948-9df7-1ee089a9d99e","Type":"ContainerDied","Data":"bca6203b28d6161e2788a99f602e46d1b57c77ccde22800de7bc856fca9ce178"} Nov 03 23:13:44.145502 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.145468 5272 scope.go:122] "RemoveContainer" containerID="4c283fa41c607e18850b5fa82e49ae59e22005ce606c9039fbc9d4312229677c" Nov 03 23:13:44.154074 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.154042 5272 scope.go:122] "RemoveContainer" containerID="117642e31b6afcf109636fda7fac6abcbd5787f4389a578e78d13307a97a6f8c" Nov 03 23:13:44.161516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.161496 5272 scope.go:122] "RemoveContainer" containerID="544a16405f75a685a95c199331984c9f152a232821be55e16cff8d8c83c05cd3" Nov 03 23:13:44.168964 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.168947 5272 scope.go:122] "RemoveContainer" containerID="8c0d8d79143989c4508bcecbe26fd08ec634eb923772ee93b1c9fd2c230d09ec" Nov 03 23:13:44.176284 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.176266 5272 scope.go:122] "RemoveContainer" containerID="416cf343b78f49f40475e5a380d5633f71c43007191ca5bd228295c79c955acf" Nov 03 23:13:44.183151 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.183131 5272 scope.go:122] "RemoveContainer" containerID="6418502d62faa490b0056469c8c0ff15e1e31888ef86c6da2599b0f96e3e7ad0" Nov 03 23:13:44.593615 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.593589 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:44.711092 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.711062 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-openshift-ca\") pod \"16ef3981-0023-4948-9df7-1ee089a9d99e\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " Nov 03 23:13:44.711262 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.711152 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ef3981-0023-4948-9df7-1ee089a9d99e-kube-api-access-mz4nr\" (UniqueName: \"kubernetes.io/projected/16ef3981-0023-4948-9df7-1ee089a9d99e-kube-api-access-mz4nr\") pod \"16ef3981-0023-4948-9df7-1ee089a9d99e\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " Nov 03 23:13:44.711262 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.711179 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-xks-ca\") pod \"16ef3981-0023-4948-9df7-1ee089a9d99e\" (UID: \"16ef3981-0023-4948-9df7-1ee089a9d99e\") " Nov 03 23:13:44.711529 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.711500 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-openshift-ca" pod "16ef3981-0023-4948-9df7-1ee089a9d99e" (UID: "16ef3981-0023-4948-9df7-1ee089a9d99e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:44.713318 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.713290 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-xks-ca" pod "16ef3981-0023-4948-9df7-1ee089a9d99e" (UID: "16ef3981-0023-4948-9df7-1ee089a9d99e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:44.713426 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.713384 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ef3981-0023-4948-9df7-1ee089a9d99e-kube-api-access-mz4nr" pod "16ef3981-0023-4948-9df7-1ee089a9d99e" (UID: "16ef3981-0023-4948-9df7-1ee089a9d99e"). InnerVolumeSpecName "kube-api-access-mz4nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:44.812140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.812103 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4nr\" (UniqueName: \"kubernetes.io/projected/16ef3981-0023-4948-9df7-1ee089a9d99e-kube-api-access-mz4nr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:44.812140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.812135 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:44.812140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:44.812147 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16ef3981-0023-4948-9df7-1ee089a9d99e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:45.470505 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:45.470473 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" Nov 03 23:13:45.470656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:45.470478 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z" event={"ID":"16ef3981-0023-4948-9df7-1ee089a9d99e","Type":"ContainerDied","Data":"6587654bf24167bc51489384b2b3224f621b5c7b9e98fec3b7dd8cdedff6b31d"} Nov 03 23:13:45.470656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:45.470623 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6587654bf24167bc51489384b2b3224f621b5c7b9e98fec3b7dd8cdedff6b31d" Nov 03 23:13:54.071468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:13:54.071434 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:13:54.500064 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:13:54.499986 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:14:02.059186 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:02.059135 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v"] Nov 03 23:14:02.103248 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:02.103211 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb4c8ef-lxr6v"] Nov 03 23:14:03.491228 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:03.491195 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb9a729-bf6f-4978-bf3b-a1bac42b9efe" path="/var/lib/kubelet/pods/8bb9a729-bf6f-4978-bf3b-a1bac42b9efe/volumes" Nov 03 23:14:04.976429 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:04.976392 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:14:05.067653 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.067623 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn"] Nov 03 23:14:05.067992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.067980 5272 cpu_manager.go:414] "containerMap: removing container" podUID="16ef3981-0023-4948-9df7-1ee089a9d99e" containerName="revoke-keys" Nov 03 23:14:05.068052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.067993 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ef3981-0023-4948-9df7-1ee089a9d99e" containerName="revoke-keys" Nov 03 23:14:05.068091 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.068080 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ef3981-0023-4948-9df7-1ee089a9d99e" containerName="revoke-keys" Nov 03 23:14:05.072399 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.072379 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.140672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.140642 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.140847 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.140678 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.140847 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.140704 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.140994 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.140847 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.140994 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.140970 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.141104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.141013 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.141104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.141041 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.141104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.141088 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lfss\" (UniqueName: \"kubernetes.io/projected/1461ab8f-e9cf-409b-ad59-2bc90b14463f-kube-api-access-4lfss\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.141206 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.141120 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.161918 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.161893 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn"] Nov 03 23:14:05.241977 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.241899 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242023 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242052 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242075 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242100 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lfss\" (UniqueName: \"kubernetes.io/projected/1461ab8f-e9cf-409b-ad59-2bc90b14463f-kube-api-access-4lfss\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242128 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242214 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242238 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242255 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242595 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242510 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242595 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242532 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.242898 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242870 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.243042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242952 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.243042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.242965 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.245045 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.245015 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.245256 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.245237 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.299134 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.299107 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1461ab8f-e9cf-409b-ad59-2bc90b14463f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.299259 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.299241 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lfss\" (UniqueName: \"kubernetes.io/projected/1461ab8f-e9cf-409b-ad59-2bc90b14463f-kube-api-access-4lfss\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wkwkn\" (UID: \"1461ab8f-e9cf-409b-ad59-2bc90b14463f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.384990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.384969 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:05.808965 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.808930 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn"] Nov 03 23:14:05.815995 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:14:05.815969 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1461ab8f_e9cf_409b_ad59_2bc90b14463f.slice/crio-7e8684d1b2458426ec0ecdb889c73f3a73783f9501d059deea48859d85875c01 WatchSource:0}: Error finding container 7e8684d1b2458426ec0ecdb889c73f3a73783f9501d059deea48859d85875c01: Status 404 returned error can't find the container with id 7e8684d1b2458426ec0ecdb889c73f3a73783f9501d059deea48859d85875c01 Nov 03 23:14:05.818127 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.818095 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:14:05.818190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.818160 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:14:05.818235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:05.818192 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:14:06.632885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:06.632838 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" event={"ID":"1461ab8f-e9cf-409b-ad59-2bc90b14463f","Type":"ContainerStarted","Data":"313b09581edba55073c5250955f88ea4b89703350ed3d2690a7f252f7d8a8724"} Nov 03 23:14:06.633381 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:06.632901 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" event={"ID":"1461ab8f-e9cf-409b-ad59-2bc90b14463f","Type":"ContainerStarted","Data":"7e8684d1b2458426ec0ecdb889c73f3a73783f9501d059deea48859d85875c01"} Nov 03 23:14:06.793232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:06.793179 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podStartSLOduration=1.7931650989999999 podStartE2EDuration="1.793165099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:14:06.791708712 +0000 UTC m=+3027.414017020" watchObservedRunningTime="2026-11-03 23:14:06.793165099 +0000 UTC m=+3027.415473407" Nov 03 23:14:07.386014 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:07.385969 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:07.387445 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:07.387420 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 23:14:07.387551 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:07.387475 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 23:14:08.386353 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:08.386313 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 23:14:08.386875 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:08.386377 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 23:14:09.385894 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:09.385858 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 23:14:09.386099 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:09.385919 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 23:14:10.386215 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:10.386169 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 23:14:10.386625 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:10.386249 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 23:14:11.385680 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:11.385640 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 23:14:11.385892 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:11.385705 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 23:14:12.386358 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.386322 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 03 23:14:12.386786 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.386388 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 03 23:14:12.846213 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.846177 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd"] Nov 03 23:14:12.849655 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.849632 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:12.898335 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.898298 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:12.924226 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.924194 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzd25\" (UniqueName: \"kubernetes.io/projected/846c8abe-4881-45fe-bc62-33012b1d1662-kube-api-access-bzd25\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:12.924376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.924317 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:12.924425 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.924371 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:12.944931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.944901 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:14:12.944931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.944904 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:14:12.945121 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.944939 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:14:12.946679 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:12.946661 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd"] Nov 03 23:14:13.025638 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.025601 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzd25\" (UniqueName: \"kubernetes.io/projected/846c8abe-4881-45fe-bc62-33012b1d1662-kube-api-access-bzd25\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:13.025831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.025687 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:13.025831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.025725 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:13.026438 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.026412 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:13.027984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.027963 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:13.083981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.083955 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzd25\" (UniqueName: \"kubernetes.io/projected/846c8abe-4881-45fe-bc62-33012b1d1662-kube-api-access-bzd25\") pod \"maas-api-revoke-sub-2e1f5b5f-52vxd\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:13.166838 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.166730 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:13.543033 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.542579 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd"] Nov 03 23:14:13.554440 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:14:13.554412 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846c8abe_4881_45fe_bc62_33012b1d1662.slice/crio-05153775c94cd53e705fd313a1add2417c1de8e0095497badaa278b1072b416f WatchSource:0}: Error finding container 05153775c94cd53e705fd313a1add2417c1de8e0095497badaa278b1072b416f: Status 404 returned error can't find the container with id 05153775c94cd53e705fd313a1add2417c1de8e0095497badaa278b1072b416f Nov 03 23:14:13.691659 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:13.691613 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" event={"ID":"846c8abe-4881-45fe-bc62-33012b1d1662","Type":"ContainerStarted","Data":"05153775c94cd53e705fd313a1add2417c1de8e0095497badaa278b1072b416f"} Nov 03 23:14:14.145746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:14.145654 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:14.146104 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:14.146036 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:14.146896 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:14.146873 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" Nov 03 23:14:14.698464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:14.698374 5272 generic.go:356] "Generic (PLEG): container finished" podID="846c8abe-4881-45fe-bc62-33012b1d1662" containerID="88057c8d88559080f012a7041b35d91e7dbed1043021e372c0eea071a32367b8" exitCode=0 Nov 03 23:14:14.698464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:14.698448 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" event={"ID":"846c8abe-4881-45fe-bc62-33012b1d1662","Type":"ContainerDied","Data":"88057c8d88559080f012a7041b35d91e7dbed1043021e372c0eea071a32367b8"} Nov 03 23:14:15.843661 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:15.843635 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:15.953273 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:15.953242 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-openshift-ca\") pod \"846c8abe-4881-45fe-bc62-33012b1d1662\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " Nov 03 23:14:15.953441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:15.953286 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846c8abe-4881-45fe-bc62-33012b1d1662-kube-api-access-bzd25\" (UniqueName: \"kubernetes.io/projected/846c8abe-4881-45fe-bc62-33012b1d1662-kube-api-access-bzd25\") pod \"846c8abe-4881-45fe-bc62-33012b1d1662\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " Nov 03 23:14:15.953441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:15.953308 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-xks-ca\") pod \"846c8abe-4881-45fe-bc62-33012b1d1662\" (UID: \"846c8abe-4881-45fe-bc62-33012b1d1662\") " Nov 03 23:14:15.953831 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:15.953744 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-openshift-ca" pod "846c8abe-4881-45fe-bc62-33012b1d1662" (UID: "846c8abe-4881-45fe-bc62-33012b1d1662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:15.955587 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:15.955547 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-xks-ca" pod "846c8abe-4881-45fe-bc62-33012b1d1662" (UID: "846c8abe-4881-45fe-bc62-33012b1d1662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:15.955733 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:15.955650 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846c8abe-4881-45fe-bc62-33012b1d1662-kube-api-access-bzd25" pod "846c8abe-4881-45fe-bc62-33012b1d1662" (UID: "846c8abe-4881-45fe-bc62-33012b1d1662"). InnerVolumeSpecName "kube-api-access-bzd25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:16.054885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.054843 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:16.054885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.054878 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzd25\" (UniqueName: \"kubernetes.io/projected/846c8abe-4881-45fe-bc62-33012b1d1662-kube-api-access-bzd25\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:16.054885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.054891 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846c8abe-4881-45fe-bc62-33012b1d1662-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:16.071083 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.071045 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp"] Nov 03 23:14:16.074226 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.074199 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc98f9c-hh7jp"] Nov 03 23:14:16.709705 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.709675 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" Nov 03 23:14:16.709905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.709718 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd" event={"ID":"846c8abe-4881-45fe-bc62-33012b1d1662","Type":"ContainerDied","Data":"05153775c94cd53e705fd313a1add2417c1de8e0095497badaa278b1072b416f"} Nov 03 23:14:16.709905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:16.709768 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05153775c94cd53e705fd313a1add2417c1de8e0095497badaa278b1072b416f" Nov 03 23:14:17.491707 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:17.491674 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796a1c3a-54ab-4f70-8574-c499b89f0d24" path="/var/lib/kubelet/pods/796a1c3a-54ab-4f70-8574-c499b89f0d24/volumes" Nov 03 23:14:24.499743 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:14:24.499700 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:14:25.326567 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.326530 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp"] Nov 03 23:14:25.327004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.326989 5272 cpu_manager.go:414] "containerMap: removing container" podUID="846c8abe-4881-45fe-bc62-33012b1d1662" containerName="revoke-keys" Nov 03 23:14:25.327060 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.327006 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846c8abe-4881-45fe-bc62-33012b1d1662" containerName="revoke-keys" Nov 03 23:14:25.327098 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.327080 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="846c8abe-4881-45fe-bc62-33012b1d1662" containerName="revoke-keys" Nov 03 23:14:25.330349 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.330327 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.425822 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.425773 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:14:25.426030 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.426011 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:14:25.426132 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.426011 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:25.426198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.426169 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:14:25.427718 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.427697 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp"] Nov 03 23:14:25.443611 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.443580 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.443611 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.443612 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm9fz\" (UniqueName: \"kubernetes.io/projected/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-kube-api-access-hm9fz\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.443853 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.443648 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.544550 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.544510 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.545113 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.544736 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.545113 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.544774 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm9fz\" (UniqueName: \"kubernetes.io/projected/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-kube-api-access-hm9fz\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.545480 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.545459 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.547040 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.547017 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.602264 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.602191 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm9fz\" (UniqueName: \"kubernetes.io/projected/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-kube-api-access-hm9fz\") pod \"maas-api-revoke-sub-ec843018-lfdjp\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.643128 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.643095 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:25.689335 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:25.689303 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:14:26.014516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:26.014431 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp"] Nov 03 23:14:26.021420 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:14:26.021391 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7a7c83_6eb2_4531_a1a7_86aaf371edc5.slice/crio-74220f3123112278d48c490a4a1787242003f8b0b3abc13864696091a6f7934d WatchSource:0}: Error finding container 74220f3123112278d48c490a4a1787242003f8b0b3abc13864696091a6f7934d: Status 404 returned error can't find the container with id 74220f3123112278d48c490a4a1787242003f8b0b3abc13864696091a6f7934d Nov 03 23:14:26.761396 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:26.761289 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" event={"ID":"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5","Type":"ContainerStarted","Data":"85aa36967dcf53f940ea228dbcedd3f47bc823cb0c5c76931c82a49fe697e5ce"} Nov 03 23:14:26.761396 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:26.761363 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" event={"ID":"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5","Type":"ContainerStarted","Data":"74220f3123112278d48c490a4a1787242003f8b0b3abc13864696091a6f7934d"} Nov 03 23:14:26.870887 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:26.870832 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" podStartSLOduration=1.870816706 podStartE2EDuration="1.870816706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:14:26.869896438 +0000 UTC m=+3047.492204747" watchObservedRunningTime="2026-11-03 23:14:26.870816706 +0000 UTC m=+3047.493125011" Nov 03 23:14:27.768762 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:27.768728 5272 generic.go:356] "Generic (PLEG): container finished" podID="5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" containerID="85aa36967dcf53f940ea228dbcedd3f47bc823cb0c5c76931c82a49fe697e5ce" exitCode=0 Nov 03 23:14:27.769208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:27.768852 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" event={"ID":"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5","Type":"ContainerDied","Data":"85aa36967dcf53f940ea228dbcedd3f47bc823cb0c5c76931c82a49fe697e5ce"} Nov 03 23:14:28.904702 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.904676 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:28.978919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.978889 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-openshift-ca\") pod \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " Nov 03 23:14:28.979099 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.979029 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-kube-api-access-hm9fz\" (UniqueName: \"kubernetes.io/projected/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-kube-api-access-hm9fz\") pod \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " Nov 03 23:14:28.979099 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.979062 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-xks-ca\") pod \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\" (UID: \"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5\") " Nov 03 23:14:28.979263 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.979241 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-openshift-ca" pod "5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" (UID: "5e7a7c83-6eb2-4531-a1a7-86aaf371edc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:28.979378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.979365 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:28.981160 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.981130 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-kube-api-access-hm9fz" pod "5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" (UID: "5e7a7c83-6eb2-4531-a1a7-86aaf371edc5"). InnerVolumeSpecName "kube-api-access-hm9fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:28.981160 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:28.981145 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-xks-ca" pod "5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" (UID: "5e7a7c83-6eb2-4531-a1a7-86aaf371edc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:29.080868 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:29.080831 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm9fz\" (UniqueName: \"kubernetes.io/projected/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-kube-api-access-hm9fz\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:29.080868 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:29.080864 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:14:29.779649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:29.779560 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" Nov 03 23:14:29.779649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:29.779589 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp" event={"ID":"5e7a7c83-6eb2-4531-a1a7-86aaf371edc5","Type":"ContainerDied","Data":"74220f3123112278d48c490a4a1787242003f8b0b3abc13864696091a6f7934d"} Nov 03 23:14:29.779649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:29.779644 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74220f3123112278d48c490a4a1787242003f8b0b3abc13864696091a6f7934d" Nov 03 23:14:42.561129 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:42.561090 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:14:45.197829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:45.196964 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:14:45.197829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:14:45.197085 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:14:54.499225 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:14:54.499142 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:15:00.165471 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.165435 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp"] Nov 03 23:15:00.166031 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.165849 5272 cpu_manager.go:414] "containerMap: removing container" podUID="5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" containerName="revoke-keys" Nov 03 23:15:00.166031 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.165863 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" containerName="revoke-keys" Nov 03 23:15:00.166031 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.165951 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" containerName="revoke-keys" Nov 03 23:15:00.170270 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.170243 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.204988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.204954 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6zt\" (UniqueName: \"kubernetes.io/projected/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-kube-api-access-lc6zt\") pod \"maas-api-key-cleanup-29895795-74wnp\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.205175 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.205023 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-74wnp\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.220933 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.220887 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:15:00.267265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.267233 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:15:00.267448 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.267339 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:15:00.269006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.268983 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp"] Nov 03 23:15:00.305829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.305774 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6zt\" (UniqueName: \"kubernetes.io/projected/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-kube-api-access-lc6zt\") pod \"maas-api-key-cleanup-29895795-74wnp\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.306030 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.305861 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-74wnp\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.306482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.306459 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-74wnp\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.363222 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.363192 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6zt\" (UniqueName: \"kubernetes.io/projected/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-kube-api-access-lc6zt\") pod \"maas-api-key-cleanup-29895795-74wnp\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.483927 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.483831 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:00.853019 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.852976 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp"] Nov 03 23:15:00.860197 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:15:00.860170 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3fda6b_36ad_440d_83d4_a27bc9b5d64d.slice/crio-2e7bd0cf20c5b3b33424287f328f3a222d1eb0fd09449bf67ee042a040054607 WatchSource:0}: Error finding container 2e7bd0cf20c5b3b33424287f328f3a222d1eb0fd09449bf67ee042a040054607: Status 404 returned error can't find the container with id 2e7bd0cf20c5b3b33424287f328f3a222d1eb0fd09449bf67ee042a040054607 Nov 03 23:15:00.959909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:00.959874 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" event={"ID":"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d","Type":"ContainerStarted","Data":"2e7bd0cf20c5b3b33424287f328f3a222d1eb0fd09449bf67ee042a040054607"} Nov 03 23:15:01.973033 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:01.972937 5272 generic.go:356] "Generic (PLEG): container finished" podID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerID="ba0ab57212d50f0e20356e2aa06cf15edaa63ce5ed0340863662a1107897ef01" exitCode=60 Nov 03 23:15:01.973434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:01.973017 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" event={"ID":"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d","Type":"ContainerDied","Data":"ba0ab57212d50f0e20356e2aa06cf15edaa63ce5ed0340863662a1107897ef01"} Nov 03 23:15:01.973434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:01.973251 5272 scope.go:122] "RemoveContainer" containerID="ba0ab57212d50f0e20356e2aa06cf15edaa63ce5ed0340863662a1107897ef01" Nov 03 23:15:02.980790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:02.980756 5272 generic.go:356] "Generic (PLEG): container finished" podID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerID="31dcf9b7c4091c29b240819ef1d43d508fddbcde90628e4b5edb436cb5b741fd" exitCode=60 Nov 03 23:15:02.980790 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:02.980839 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" event={"ID":"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d","Type":"ContainerDied","Data":"31dcf9b7c4091c29b240819ef1d43d508fddbcde90628e4b5edb436cb5b741fd"} Nov 03 23:15:02.981408 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:02.980905 5272 scope.go:122] "RemoveContainer" containerID="ba0ab57212d50f0e20356e2aa06cf15edaa63ce5ed0340863662a1107897ef01" Nov 03 23:15:02.981408 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:02.981161 5272 scope.go:122] "RemoveContainer" containerID="31dcf9b7c4091c29b240819ef1d43d508fddbcde90628e4b5edb436cb5b741fd" Nov 03 23:15:02.981585 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:15:02.981450 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-74wnp_odh-ai-gateway-infra(4d3fda6b-36ad-440d-83d4-a27bc9b5d64d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" Nov 03 23:15:03.988491 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:03.988457 5272 scope.go:122] "RemoveContainer" containerID="31dcf9b7c4091c29b240819ef1d43d508fddbcde90628e4b5edb436cb5b741fd" Nov 03 23:15:03.988918 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:15:03.988705 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-74wnp_odh-ai-gateway-infra(4d3fda6b-36ad-440d-83d4-a27bc9b5d64d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" Nov 03 23:15:13.581436 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:13.581388 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:15:14.486635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:14.486585 5272 scope.go:122] "RemoveContainer" containerID="31dcf9b7c4091c29b240819ef1d43d508fddbcde90628e4b5edb436cb5b741fd" Nov 03 23:15:15.054829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:15.051371 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" event={"ID":"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d","Type":"ContainerStarted","Data":"30ee7bd66d163d37af24869f4867ceb078c3cd19c0dff8395a404570b608091c"} Nov 03 23:15:16.069829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:16.067990 5272 generic.go:356] "Generic (PLEG): container finished" podID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerID="30ee7bd66d163d37af24869f4867ceb078c3cd19c0dff8395a404570b608091c" exitCode=60 Nov 03 23:15:16.069829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:16.068140 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" event={"ID":"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d","Type":"ContainerDied","Data":"30ee7bd66d163d37af24869f4867ceb078c3cd19c0dff8395a404570b608091c"} Nov 03 23:15:16.069829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:16.068218 5272 scope.go:122] "RemoveContainer" containerID="31dcf9b7c4091c29b240819ef1d43d508fddbcde90628e4b5edb436cb5b741fd" Nov 03 23:15:16.069829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:16.068840 5272 scope.go:122] "RemoveContainer" containerID="30ee7bd66d163d37af24869f4867ceb078c3cd19c0dff8395a404570b608091c" Nov 03 23:15:16.069829 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:15:16.069204 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-74wnp_odh-ai-gateway-infra(4d3fda6b-36ad-440d-83d4-a27bc9b5d64d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" Nov 03 23:15:16.220830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:16.218709 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp"] Nov 03 23:15:17.200174 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:17.200143 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:17.320327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:17.320294 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-maas-api-service-ca\") pod \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " Nov 03 23:15:17.320525 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:17.320459 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-kube-api-access-lc6zt\" (UniqueName: \"kubernetes.io/projected/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-kube-api-access-lc6zt\") pod \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\" (UID: \"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d\") " Nov 03 23:15:17.320669 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:17.320645 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-maas-api-service-ca" pod "4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" (UID: "4d3fda6b-36ad-440d-83d4-a27bc9b5d64d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:17.320751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:17.320735 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:17.322470 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:17.322444 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-kube-api-access-lc6zt" pod "4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" (UID: "4d3fda6b-36ad-440d-83d4-a27bc9b5d64d"). InnerVolumeSpecName "kube-api-access-lc6zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:17.422198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:17.422082 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6zt\" (UniqueName: \"kubernetes.io/projected/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d-kube-api-access-lc6zt\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:18.079654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:18.079624 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" Nov 03 23:15:18.079873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:18.079658 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp" event={"ID":"4d3fda6b-36ad-440d-83d4-a27bc9b5d64d","Type":"ContainerDied","Data":"2e7bd0cf20c5b3b33424287f328f3a222d1eb0fd09449bf67ee042a040054607"} Nov 03 23:15:18.079873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:18.079730 5272 scope.go:122] "RemoveContainer" containerID="30ee7bd66d163d37af24869f4867ceb078c3cd19c0dff8395a404570b608091c" Nov 03 23:15:18.233163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:18.233129 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp"] Nov 03 23:15:18.279304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:18.279273 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-74wnp"] Nov 03 23:15:19.495658 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:19.495618 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" path="/var/lib/kubelet/pods/4d3fda6b-36ad-440d-83d4-a27bc9b5d64d/volumes" Nov 03 23:15:24.358608 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:24.358572 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:15:24.500367 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:15:24.500331 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:15:37.066330 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:37.066292 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:15:44.259028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:44.258994 5272 scope.go:122] "RemoveContainer" containerID="9ef02425f21edff73f406dd0abdf1f15651274bf848e8da579afcc5bcde260ff" Nov 03 23:15:44.267351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:44.267328 5272 scope.go:122] "RemoveContainer" containerID="c8841f7031d9b88b1069f5d51a329f18bf4766510bc115c105faebcb0f2c33ca" Nov 03 23:15:44.275428 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:44.275397 5272 scope.go:122] "RemoveContainer" containerID="1c3db130f126db6b4635700d866a4dbdf6f89c071358825b469cbbe7b2fcfd5e" Nov 03 23:15:49.286431 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.286396 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq"] Nov 03 23:15:49.287001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.286887 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:15:49.287001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.286903 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:15:49.287001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.286924 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:15:49.287001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.286929 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:15:49.287001 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.287002 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:15:49.287192 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.287014 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:15:49.287192 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.287020 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:15:49.291911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.291885 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.342514 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.342479 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:15:49.381697 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.381664 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbt2\" (UniqueName: \"kubernetes.io/projected/38199f36-7b56-446d-b872-e59e48a1b65c-kube-api-access-6dbt2\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.381697 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.381700 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.381937 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.381881 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.389306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.389280 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:15:49.389306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.389292 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:15:49.389306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.389312 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:15:49.391222 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.391203 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq"] Nov 03 23:15:49.483051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.483015 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.483231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.483149 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.483231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.483200 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbt2\" (UniqueName: \"kubernetes.io/projected/38199f36-7b56-446d-b872-e59e48a1b65c-kube-api-access-6dbt2\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.483762 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.483739 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.485405 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.485377 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.540511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.540443 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbt2\" (UniqueName: \"kubernetes.io/projected/38199f36-7b56-446d-b872-e59e48a1b65c-kube-api-access-6dbt2\") pod \"maas-api-revoke-sub-ae594979-5bstq\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.604755 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.604722 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:49.976467 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:49.976362 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq"] Nov 03 23:15:49.983532 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:15:49.983506 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38199f36_7b56_446d_b872_e59e48a1b65c.slice/crio-cf802ec8c133957f73625f692e8adc695312dbbef5e30a3a2a9d7e8ae177be1b WatchSource:0}: Error finding container cf802ec8c133957f73625f692e8adc695312dbbef5e30a3a2a9d7e8ae177be1b: Status 404 returned error can't find the container with id cf802ec8c133957f73625f692e8adc695312dbbef5e30a3a2a9d7e8ae177be1b Nov 03 23:15:50.281406 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:50.281305 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" event={"ID":"38199f36-7b56-446d-b872-e59e48a1b65c","Type":"ContainerStarted","Data":"49741dbd274d6e70156f0383ca44cf27972c6978fa2f9d8eec13ab753c5e1afc"} Nov 03 23:15:50.281406 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:50.281373 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" event={"ID":"38199f36-7b56-446d-b872-e59e48a1b65c","Type":"ContainerStarted","Data":"cf802ec8c133957f73625f692e8adc695312dbbef5e30a3a2a9d7e8ae177be1b"} Nov 03 23:15:50.391087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:50.391034 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" podStartSLOduration=1.391019427 podStartE2EDuration="1.391019427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:50.389955176 +0000 UTC m=+3131.012263481" watchObservedRunningTime="2026-11-03 23:15:50.391019427 +0000 UTC m=+3131.013327735" Nov 03 23:15:51.287656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:51.287623 5272 generic.go:356] "Generic (PLEG): container finished" podID="38199f36-7b56-446d-b872-e59e48a1b65c" containerID="49741dbd274d6e70156f0383ca44cf27972c6978fa2f9d8eec13ab753c5e1afc" exitCode=0 Nov 03 23:15:51.287856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:51.287706 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" event={"ID":"38199f36-7b56-446d-b872-e59e48a1b65c","Type":"ContainerDied","Data":"49741dbd274d6e70156f0383ca44cf27972c6978fa2f9d8eec13ab753c5e1afc"} Nov 03 23:15:52.426645 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.426614 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:52.512074 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.512040 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38199f36-7b56-446d-b872-e59e48a1b65c-kube-api-access-6dbt2\" (UniqueName: \"kubernetes.io/projected/38199f36-7b56-446d-b872-e59e48a1b65c-kube-api-access-6dbt2\") pod \"38199f36-7b56-446d-b872-e59e48a1b65c\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " Nov 03 23:15:52.512279 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.512127 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-xks-ca\") pod \"38199f36-7b56-446d-b872-e59e48a1b65c\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " Nov 03 23:15:52.512279 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.512157 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-openshift-ca\") pod \"38199f36-7b56-446d-b872-e59e48a1b65c\" (UID: \"38199f36-7b56-446d-b872-e59e48a1b65c\") " Nov 03 23:15:52.512580 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.512557 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-openshift-ca" pod "38199f36-7b56-446d-b872-e59e48a1b65c" (UID: "38199f36-7b56-446d-b872-e59e48a1b65c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:52.514290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.514266 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38199f36-7b56-446d-b872-e59e48a1b65c-kube-api-access-6dbt2" pod "38199f36-7b56-446d-b872-e59e48a1b65c" (UID: "38199f36-7b56-446d-b872-e59e48a1b65c"). InnerVolumeSpecName "kube-api-access-6dbt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:52.514417 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.514316 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-xks-ca" pod "38199f36-7b56-446d-b872-e59e48a1b65c" (UID: "38199f36-7b56-446d-b872-e59e48a1b65c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:52.613498 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.613407 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:52.613498 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.613438 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38199f36-7b56-446d-b872-e59e48a1b65c-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:52.613498 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:52.613448 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dbt2\" (UniqueName: \"kubernetes.io/projected/38199f36-7b56-446d-b872-e59e48a1b65c-kube-api-access-6dbt2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:15:53.299107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:53.299068 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" event={"ID":"38199f36-7b56-446d-b872-e59e48a1b65c","Type":"ContainerDied","Data":"cf802ec8c133957f73625f692e8adc695312dbbef5e30a3a2a9d7e8ae177be1b"} Nov 03 23:15:53.299107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:53.299092 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq" Nov 03 23:15:53.299334 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:15:53.299153 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf802ec8c133957f73625f692e8adc695312dbbef5e30a3a2a9d7e8ae177be1b" Nov 03 23:15:54.500061 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:15:54.500024 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:16:00.054993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:00.054952 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf"] Nov 03 23:16:00.101386 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:00.101349 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf66932-8pnsf"] Nov 03 23:16:01.491713 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:01.491674 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723d2056-0ebe-4a91-9168-dc7d58211f48" path="/var/lib/kubelet/pods/723d2056-0ebe-4a91-9168-dc7d58211f48/volumes" Nov 03 23:16:02.181610 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:02.181571 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:16:11.851019 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:11.850984 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:16:17.877495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.877458 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5"] Nov 03 23:16:17.877929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.877872 5272 cpu_manager.go:414] "containerMap: removing container" podUID="38199f36-7b56-446d-b872-e59e48a1b65c" containerName="revoke-keys" Nov 03 23:16:17.877929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.877884 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38199f36-7b56-446d-b872-e59e48a1b65c" containerName="revoke-keys" Nov 03 23:16:17.877929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.877902 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:16:17.877929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.877907 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3fda6b-36ad-440d-83d4-a27bc9b5d64d" containerName="cleanup" Nov 03 23:16:17.878091 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.877979 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="38199f36-7b56-446d-b872-e59e48a1b65c" containerName="revoke-keys" Nov 03 23:16:17.881186 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.881166 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:17.911107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.911081 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:17.911239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.911157 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:17.911292 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.911241 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6scgz\" (UniqueName: \"kubernetes.io/projected/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-kube-api-access-6scgz\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:17.930322 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.930298 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:16:17.930322 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.930302 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:16:17.930597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.930576 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:16:17.976997 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.976959 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:16:17.978763 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:17.978736 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5"] Nov 03 23:16:18.012193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.012165 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:18.012317 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.012217 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6scgz\" (UniqueName: \"kubernetes.io/projected/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-kube-api-access-6scgz\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:18.012317 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.012262 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:18.013009 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.012989 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:18.014581 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.014563 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:18.070281 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.070247 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6scgz\" (UniqueName: \"kubernetes.io/projected/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-kube-api-access-6scgz\") pod \"maas-api-revoke-sub-a4aef7d7-829w5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:18.193791 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.193714 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:18.562909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:18.562874 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5"] Nov 03 23:16:18.570162 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:16:18.570135 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5a3cf5_fea3_455b_9a0a_ad466bbcc2b5.slice/crio-19a3a6c9a6c015e3ac905f30dbffc9089b9561be6b65c05ce000cc2a383834e3 WatchSource:0}: Error finding container 19a3a6c9a6c015e3ac905f30dbffc9089b9561be6b65c05ce000cc2a383834e3: Status 404 returned error can't find the container with id 19a3a6c9a6c015e3ac905f30dbffc9089b9561be6b65c05ce000cc2a383834e3 Nov 03 23:16:19.444774 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:19.444742 5272 generic.go:356] "Generic (PLEG): container finished" podID="be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" containerID="6869d862b0c3f47daa2f4c3888cdfd48ab7521b5baef9a09c762a9bc31c76e0d" exitCode=0 Nov 03 23:16:19.445146 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:19.444828 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" event={"ID":"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5","Type":"ContainerDied","Data":"6869d862b0c3f47daa2f4c3888cdfd48ab7521b5baef9a09c762a9bc31c76e0d"} Nov 03 23:16:19.445146 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:19.444879 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" event={"ID":"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5","Type":"ContainerStarted","Data":"19a3a6c9a6c015e3ac905f30dbffc9089b9561be6b65c05ce000cc2a383834e3"} Nov 03 23:16:20.580395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.580358 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:20.634922 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.634884 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-openshift-ca\") pod \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " Nov 03 23:16:20.635128 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.634941 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-xks-ca\") pod \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " Nov 03 23:16:20.635128 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.634996 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-kube-api-access-6scgz\" (UniqueName: \"kubernetes.io/projected/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-kube-api-access-6scgz\") pod \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\" (UID: \"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5\") " Nov 03 23:16:20.635354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.635329 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-openshift-ca" pod "be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" (UID: "be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:16:20.637190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.637167 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-xks-ca" pod "be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" (UID: "be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:16:20.637285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.637198 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-kube-api-access-6scgz" pod "be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" (UID: "be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5"). InnerVolumeSpecName "kube-api-access-6scgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:16:20.736460 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.736367 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6scgz\" (UniqueName: \"kubernetes.io/projected/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-kube-api-access-6scgz\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:16:20.736460 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.736409 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:16:20.736460 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:20.736420 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:16:21.456145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:21.456114 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" Nov 03 23:16:21.456304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:21.456143 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5" event={"ID":"be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5","Type":"ContainerDied","Data":"19a3a6c9a6c015e3ac905f30dbffc9089b9561be6b65c05ce000cc2a383834e3"} Nov 03 23:16:21.456304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:21.456220 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19a3a6c9a6c015e3ac905f30dbffc9089b9561be6b65c05ce000cc2a383834e3" Nov 03 23:16:24.500064 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:16:24.500031 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:16:30.772891 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:30.772857 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:16:43.661698 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:43.661659 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:16:44.304764 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:44.304729 5272 scope.go:122] "RemoveContainer" containerID="3d201221d0a0bcdf622079c4d79fcfe9e684dfbba9b19fc07f18792cfe28e597" Nov 03 23:16:48.581554 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.581509 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72"] Nov 03 23:16:48.582131 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.582108 5272 cpu_manager.go:414] "containerMap: removing container" podUID="be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" containerName="revoke-keys" Nov 03 23:16:48.582217 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.582134 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" containerName="revoke-keys" Nov 03 23:16:48.582283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.582237 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" containerName="revoke-keys" Nov 03 23:16:48.586795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.586770 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.636309 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.636275 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:16:48.636462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.636318 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:16:48.636509 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.636484 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:16:48.682572 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.682540 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:16:48.684342 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.684321 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72"] Nov 03 23:16:48.757221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.757183 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.757221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.757224 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zn55\" (UniqueName: \"kubernetes.io/projected/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-kube-api-access-5zn55\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.757427 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.757246 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.858772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.858697 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.858772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.858732 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zn55\" (UniqueName: \"kubernetes.io/projected/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-kube-api-access-5zn55\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.858772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.858755 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.859449 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.859417 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.861223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.861200 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:48.916482 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:48.916451 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zn55\" (UniqueName: \"kubernetes.io/projected/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-kube-api-access-5zn55\") pod \"maas-api-revoke-sub-79805ca9-lls72\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:49.204939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:49.199273 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:49.570908 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:49.570868 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72"] Nov 03 23:16:49.578636 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:16:49.578598 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5626e8e_5fd1_4950_9d15_4ee3cddfdaae.slice/crio-42ea2f4b0c84dcf8989f2a4a933568285f8015debcee463329209ce0b11b77dd WatchSource:0}: Error finding container 42ea2f4b0c84dcf8989f2a4a933568285f8015debcee463329209ce0b11b77dd: Status 404 returned error can't find the container with id 42ea2f4b0c84dcf8989f2a4a933568285f8015debcee463329209ce0b11b77dd Nov 03 23:16:49.618574 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:49.618544 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" event={"ID":"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae","Type":"ContainerStarted","Data":"42ea2f4b0c84dcf8989f2a4a933568285f8015debcee463329209ce0b11b77dd"} Nov 03 23:16:50.625288 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:50.625250 5272 generic.go:356] "Generic (PLEG): container finished" podID="c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" containerID="acd6ebcaf7e23732a7e79e56334fa3bc3761ccf36aca1788d3fc9cd2b013b429" exitCode=0 Nov 03 23:16:50.625774 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:50.625337 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" event={"ID":"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae","Type":"ContainerDied","Data":"acd6ebcaf7e23732a7e79e56334fa3bc3761ccf36aca1788d3fc9cd2b013b429"} Nov 03 23:16:51.768762 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.768734 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:51.887981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.887897 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-openshift-ca\") pod \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " Nov 03 23:16:51.887981 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.887947 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-kube-api-access-5zn55\" (UniqueName: \"kubernetes.io/projected/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-kube-api-access-5zn55\") pod \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " Nov 03 23:16:51.888222 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.888086 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-xks-ca\") pod \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\" (UID: \"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae\") " Nov 03 23:16:51.888334 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.888304 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-openshift-ca" pod "c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" (UID: "c5626e8e-5fd1-4950-9d15-4ee3cddfdaae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:16:51.888531 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.888509 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:16:51.890153 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.890126 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-kube-api-access-5zn55" pod "c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" (UID: "c5626e8e-5fd1-4950-9d15-4ee3cddfdaae"). InnerVolumeSpecName "kube-api-access-5zn55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:16:51.891358 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.891329 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-xks-ca" pod "c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" (UID: "c5626e8e-5fd1-4950-9d15-4ee3cddfdaae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:16:51.988998 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.988963 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zn55\" (UniqueName: \"kubernetes.io/projected/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-kube-api-access-5zn55\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:16:51.988998 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:51.988994 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:16:52.636007 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:52.635963 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" event={"ID":"c5626e8e-5fd1-4950-9d15-4ee3cddfdaae","Type":"ContainerDied","Data":"42ea2f4b0c84dcf8989f2a4a933568285f8015debcee463329209ce0b11b77dd"} Nov 03 23:16:52.636189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:52.636026 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ea2f4b0c84dcf8989f2a4a933568285f8015debcee463329209ce0b11b77dd" Nov 03 23:16:52.636189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:16:52.635977 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72" Nov 03 23:16:54.499678 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:16:54.499634 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:17:03.268269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:03.268221 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:17:12.926723 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:12.926678 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n"] Nov 03 23:17:12.927324 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:12.927303 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" containerName="revoke-keys" Nov 03 23:17:12.927406 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:12.927327 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" containerName="revoke-keys" Nov 03 23:17:12.927462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:12.927446 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" containerName="revoke-keys" Nov 03 23:17:12.930952 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:12.930923 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.027477 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.027442 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:17:13.027477 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.027466 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:17:13.027477 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.027452 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:17:13.027737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.027442 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:17:13.029304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.029284 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n"] Nov 03 23:17:13.029715 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.029693 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg7kt\" (UniqueName: \"kubernetes.io/projected/bf274a12-36f2-4710-a581-b49c2b32e5d3-kube-api-access-xg7kt\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.029754 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.029727 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.029875 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.029862 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.130672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.130633 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.130888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.130727 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg7kt\" (UniqueName: \"kubernetes.io/projected/bf274a12-36f2-4710-a581-b49c2b32e5d3-kube-api-access-xg7kt\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.130888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.130749 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.131405 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.131386 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.133054 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.133022 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.188266 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.188196 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg7kt\" (UniqueName: \"kubernetes.io/projected/bf274a12-36f2-4710-a581-b49c2b32e5d3-kube-api-access-xg7kt\") pod \"maas-api-revoke-sub-0b7e8edb-slw7n\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.243628 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.243600 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:13.617218 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.617181 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n"] Nov 03 23:17:13.624316 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:17:13.624284 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf274a12_36f2_4710_a581_b49c2b32e5d3.slice/crio-e316d9a17d1a50131bc2ff3839cda3eef451a43c3f60513704a760272ccefc03 WatchSource:0}: Error finding container e316d9a17d1a50131bc2ff3839cda3eef451a43c3f60513704a760272ccefc03: Status 404 returned error can't find the container with id e316d9a17d1a50131bc2ff3839cda3eef451a43c3f60513704a760272ccefc03 Nov 03 23:17:13.760181 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:13.760141 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" event={"ID":"bf274a12-36f2-4710-a581-b49c2b32e5d3","Type":"ContainerStarted","Data":"e316d9a17d1a50131bc2ff3839cda3eef451a43c3f60513704a760272ccefc03"} Nov 03 23:17:14.766818 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:14.766777 5272 generic.go:356] "Generic (PLEG): container finished" podID="bf274a12-36f2-4710-a581-b49c2b32e5d3" containerID="e2497afa6f9fa1d26d961010b037dc1f178435a67c4c853340b5d142d42dff6e" exitCode=0 Nov 03 23:17:14.767255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:14.766868 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" event={"ID":"bf274a12-36f2-4710-a581-b49c2b32e5d3","Type":"ContainerDied","Data":"e2497afa6f9fa1d26d961010b037dc1f178435a67c4c853340b5d142d42dff6e"} Nov 03 23:17:15.901993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:15.901962 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:15.961150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:15.961113 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-xks-ca\") pod \"bf274a12-36f2-4710-a581-b49c2b32e5d3\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " Nov 03 23:17:15.961367 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:15.961172 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-openshift-ca\") pod \"bf274a12-36f2-4710-a581-b49c2b32e5d3\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " Nov 03 23:17:15.961367 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:15.961228 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf274a12-36f2-4710-a581-b49c2b32e5d3-kube-api-access-xg7kt\" (UniqueName: \"kubernetes.io/projected/bf274a12-36f2-4710-a581-b49c2b32e5d3-kube-api-access-xg7kt\") pod \"bf274a12-36f2-4710-a581-b49c2b32e5d3\" (UID: \"bf274a12-36f2-4710-a581-b49c2b32e5d3\") " Nov 03 23:17:15.961516 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:15.961488 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-openshift-ca" pod "bf274a12-36f2-4710-a581-b49c2b32e5d3" (UID: "bf274a12-36f2-4710-a581-b49c2b32e5d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:17:15.963311 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:15.963292 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-xks-ca" pod "bf274a12-36f2-4710-a581-b49c2b32e5d3" (UID: "bf274a12-36f2-4710-a581-b49c2b32e5d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:17:15.963409 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:15.963388 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf274a12-36f2-4710-a581-b49c2b32e5d3-kube-api-access-xg7kt" pod "bf274a12-36f2-4710-a581-b49c2b32e5d3" (UID: "bf274a12-36f2-4710-a581-b49c2b32e5d3"). InnerVolumeSpecName "kube-api-access-xg7kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:17:16.062473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:16.062432 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:17:16.062473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:16.062467 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf274a12-36f2-4710-a581-b49c2b32e5d3-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:17:16.062473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:16.062478 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg7kt\" (UniqueName: \"kubernetes.io/projected/bf274a12-36f2-4710-a581-b49c2b32e5d3-kube-api-access-xg7kt\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:17:16.778306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:16.778269 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" Nov 03 23:17:16.778306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:16.778293 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n" event={"ID":"bf274a12-36f2-4710-a581-b49c2b32e5d3","Type":"ContainerDied","Data":"e316d9a17d1a50131bc2ff3839cda3eef451a43c3f60513704a760272ccefc03"} Nov 03 23:17:16.778535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:16.778342 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e316d9a17d1a50131bc2ff3839cda3eef451a43c3f60513704a760272ccefc03" Nov 03 23:17:24.268555 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.268511 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq"] Nov 03 23:17:24.269157 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.268940 5272 cpu_manager.go:414] "containerMap: removing container" podUID="bf274a12-36f2-4710-a581-b49c2b32e5d3" containerName="revoke-keys" Nov 03 23:17:24.269157 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.268954 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf274a12-36f2-4710-a581-b49c2b32e5d3" containerName="revoke-keys" Nov 03 23:17:24.269157 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.269037 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf274a12-36f2-4710-a581-b49c2b32e5d3" containerName="revoke-keys" Nov 03 23:17:24.272325 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.272300 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.324350 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.324313 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:17:24.345973 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.345937 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.346139 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.346070 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxw45\" (UniqueName: \"kubernetes.io/projected/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-kube-api-access-bxw45\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.346208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.346156 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.370723 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.370685 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:17:24.370922 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.370747 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:17:24.370995 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.370920 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:17:24.372869 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.372845 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq"] Nov 03 23:17:24.446722 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.446686 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxw45\" (UniqueName: \"kubernetes.io/projected/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-kube-api-access-bxw45\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.446915 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.446735 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.446915 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.446783 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.447465 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.447432 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.449103 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.449084 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.500486 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:17:24.500450 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:17:24.504179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.504155 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxw45\" (UniqueName: \"kubernetes.io/projected/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-kube-api-access-bxw45\") pod \"maas-api-revoke-sub-3d4156f1-t44vq\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.585163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.585133 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:24.954888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:24.954788 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq"] Nov 03 23:17:24.961453 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:17:24.961428 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fab9d7_ba2a_4a7c_9339_e7b74e162d20.slice/crio-6f0ef314a8a358d2ca373aee0f74faf23d8afa0e1976251ecb899f23ef0bb652 WatchSource:0}: Error finding container 6f0ef314a8a358d2ca373aee0f74faf23d8afa0e1976251ecb899f23ef0bb652: Status 404 returned error can't find the container with id 6f0ef314a8a358d2ca373aee0f74faf23d8afa0e1976251ecb899f23ef0bb652 Nov 03 23:17:25.781848 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:25.781814 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:17:25.825195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:25.825159 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" event={"ID":"26fab9d7-ba2a-4a7c-9339-e7b74e162d20","Type":"ContainerStarted","Data":"f8fe0acead4ed2a1dcaede3921a3dbf3871b5c7240938d071840650cb07f3e0c"} Nov 03 23:17:25.825369 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:25.825208 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" event={"ID":"26fab9d7-ba2a-4a7c-9339-e7b74e162d20","Type":"ContainerStarted","Data":"6f0ef314a8a358d2ca373aee0f74faf23d8afa0e1976251ecb899f23ef0bb652"} Nov 03 23:17:25.936199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:25.936146 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" podStartSLOduration=1.9361305770000001 podStartE2EDuration="1.936130577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:25.934095087 +0000 UTC m=+3226.556403394" watchObservedRunningTime="2026-11-03 23:17:25.936130577 +0000 UTC m=+3226.558439313" Nov 03 23:17:26.850171 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:26.850118 5272 generic.go:356] "Generic (PLEG): container finished" podID="26fab9d7-ba2a-4a7c-9339-e7b74e162d20" containerID="f8fe0acead4ed2a1dcaede3921a3dbf3871b5c7240938d071840650cb07f3e0c" exitCode=0 Nov 03 23:17:26.851355 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:26.851328 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" event={"ID":"26fab9d7-ba2a-4a7c-9339-e7b74e162d20","Type":"ContainerDied","Data":"f8fe0acead4ed2a1dcaede3921a3dbf3871b5c7240938d071840650cb07f3e0c"} Nov 03 23:17:28.301225 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.301197 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:28.377756 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.377723 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-kube-api-access-bxw45\" (UniqueName: \"kubernetes.io/projected/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-kube-api-access-bxw45\") pod \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " Nov 03 23:17:28.377962 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.377939 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-xks-ca\") pod \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " Nov 03 23:17:28.378022 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.377971 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-openshift-ca\") pod \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\" (UID: \"26fab9d7-ba2a-4a7c-9339-e7b74e162d20\") " Nov 03 23:17:28.378378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.378353 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-openshift-ca" pod "26fab9d7-ba2a-4a7c-9339-e7b74e162d20" (UID: "26fab9d7-ba2a-4a7c-9339-e7b74e162d20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:17:28.379863 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.379832 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-kube-api-access-bxw45" pod "26fab9d7-ba2a-4a7c-9339-e7b74e162d20" (UID: "26fab9d7-ba2a-4a7c-9339-e7b74e162d20"). InnerVolumeSpecName "kube-api-access-bxw45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:17:28.379965 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.379943 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-xks-ca" pod "26fab9d7-ba2a-4a7c-9339-e7b74e162d20" (UID: "26fab9d7-ba2a-4a7c-9339-e7b74e162d20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:17:28.479280 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.479196 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:17:28.479280 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.479230 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:17:28.479280 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.479241 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxw45\" (UniqueName: \"kubernetes.io/projected/26fab9d7-ba2a-4a7c-9339-e7b74e162d20-kube-api-access-bxw45\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:17:28.889573 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.889540 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" Nov 03 23:17:28.889771 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.889586 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq" event={"ID":"26fab9d7-ba2a-4a7c-9339-e7b74e162d20","Type":"ContainerDied","Data":"6f0ef314a8a358d2ca373aee0f74faf23d8afa0e1976251ecb899f23ef0bb652"} Nov 03 23:17:28.889771 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:28.889647 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f0ef314a8a358d2ca373aee0f74faf23d8afa0e1976251ecb899f23ef0bb652" Nov 03 23:17:33.056537 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:33.056501 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz"] Nov 03 23:17:33.102972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:33.102941 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fnpfz"] Nov 03 23:17:33.491747 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:33.491651 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1558a0-57b5-47f0-b1c4-87ca2246a307" path="/var/lib/kubelet/pods/4a1558a0-57b5-47f0-b1c4-87ca2246a307/volumes" Nov 03 23:17:38.879385 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:38.878257 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:17:44.331571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:44.331536 5272 scope.go:122] "RemoveContainer" containerID="c54d7c84e8e20be765a820e9cd3f195bf9dd3f4b5e77a41f58cb3474b49b1fec" Nov 03 23:17:44.340231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:44.340211 5272 scope.go:122] "RemoveContainer" containerID="3f83d994538003121240d2908c4286c508f851fe31a6cf20a4ae80c85524b520" Nov 03 23:17:49.974756 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:49.974659 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:17:54.500056 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:17:54.500020 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:17:57.941360 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:57.941325 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq"] Nov 03 23:17:57.987611 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:57.987571 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t44vq"] Nov 03 23:17:58.067888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.067853 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj"] Nov 03 23:17:58.068267 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.068253 5272 cpu_manager.go:414] "containerMap: removing container" podUID="26fab9d7-ba2a-4a7c-9339-e7b74e162d20" containerName="revoke-keys" Nov 03 23:17:58.068315 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.068271 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fab9d7-ba2a-4a7c-9339-e7b74e162d20" containerName="revoke-keys" Nov 03 23:17:58.068382 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.068372 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fab9d7-ba2a-4a7c-9339-e7b74e162d20" containerName="revoke-keys" Nov 03 23:17:58.071640 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.071624 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.120984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.120950 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:17:58.121179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.121034 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj"] Nov 03 23:17:58.121179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.121166 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:17:58.121319 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.121181 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:17:58.139785 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.139757 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:17:58.149467 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.149441 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.149598 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.149503 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhglj\" (UniqueName: \"kubernetes.io/projected/dc445bee-cf8b-4c84-91ee-52771307fa9c-kube-api-access-bhglj\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.149648 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.149625 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.250691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.250602 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhglj\" (UniqueName: \"kubernetes.io/projected/dc445bee-cf8b-4c84-91ee-52771307fa9c-kube-api-access-bhglj\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.250691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.250683 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.250934 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.250747 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.251473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.251451 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.253156 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.253136 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.308582 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.308553 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhglj\" (UniqueName: \"kubernetes.io/projected/dc445bee-cf8b-4c84-91ee-52771307fa9c-kube-api-access-bhglj\") pod \"maas-api-revoke-sub-3d4156f1-b78pj\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.384234 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.384198 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:17:58.756426 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:58.756382 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj"] Nov 03 23:17:58.762842 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:17:58.762814 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc445bee_cf8b_4c84_91ee_52771307fa9c.slice/crio-bf124b5eeb9ee7c4b3569ff52eb6d9185dc4de094fc53dfa6e94c6a56baf321f WatchSource:0}: Error finding container bf124b5eeb9ee7c4b3569ff52eb6d9185dc4de094fc53dfa6e94c6a56baf321f: Status 404 returned error can't find the container with id bf124b5eeb9ee7c4b3569ff52eb6d9185dc4de094fc53dfa6e94c6a56baf321f Nov 03 23:17:59.097307 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:59.097259 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" event={"ID":"dc445bee-cf8b-4c84-91ee-52771307fa9c","Type":"ContainerStarted","Data":"183c549dbd52d60213a40fcd4241818e7782dce78c5e81e1ba511bdea662ad20"} Nov 03 23:17:59.097684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:59.097325 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" event={"ID":"dc445bee-cf8b-4c84-91ee-52771307fa9c","Type":"ContainerStarted","Data":"bf124b5eeb9ee7c4b3569ff52eb6d9185dc4de094fc53dfa6e94c6a56baf321f"} Nov 03 23:17:59.208086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:59.208025 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" podStartSLOduration=1.208006195 podStartE2EDuration="1.208006195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:59.205436867 +0000 UTC m=+3259.827745176" watchObservedRunningTime="2026-11-03 23:17:59.208006195 +0000 UTC m=+3259.830314500" Nov 03 23:17:59.491241 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:17:59.491165 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fab9d7-ba2a-4a7c-9339-e7b74e162d20" path="/var/lib/kubelet/pods/26fab9d7-ba2a-4a7c-9339-e7b74e162d20/volumes" Nov 03 23:18:00.103706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:00.103666 5272 generic.go:356] "Generic (PLEG): container finished" podID="dc445bee-cf8b-4c84-91ee-52771307fa9c" containerID="183c549dbd52d60213a40fcd4241818e7782dce78c5e81e1ba511bdea662ad20" exitCode=0 Nov 03 23:18:00.104120 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:00.103755 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" event={"ID":"dc445bee-cf8b-4c84-91ee-52771307fa9c","Type":"ContainerDied","Data":"183c549dbd52d60213a40fcd4241818e7782dce78c5e81e1ba511bdea662ad20"} Nov 03 23:18:01.242264 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.242239 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:18:01.273706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.273677 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-openshift-ca\") pod \"dc445bee-cf8b-4c84-91ee-52771307fa9c\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " Nov 03 23:18:01.273877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.273727 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-xks-ca\") pod \"dc445bee-cf8b-4c84-91ee-52771307fa9c\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " Nov 03 23:18:01.273877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.273761 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc445bee-cf8b-4c84-91ee-52771307fa9c-kube-api-access-bhglj\" (UniqueName: \"kubernetes.io/projected/dc445bee-cf8b-4c84-91ee-52771307fa9c-kube-api-access-bhglj\") pod \"dc445bee-cf8b-4c84-91ee-52771307fa9c\" (UID: \"dc445bee-cf8b-4c84-91ee-52771307fa9c\") " Nov 03 23:18:01.274093 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.274067 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-openshift-ca" pod "dc445bee-cf8b-4c84-91ee-52771307fa9c" (UID: "dc445bee-cf8b-4c84-91ee-52771307fa9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:01.275785 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.275762 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc445bee-cf8b-4c84-91ee-52771307fa9c-kube-api-access-bhglj" pod "dc445bee-cf8b-4c84-91ee-52771307fa9c" (UID: "dc445bee-cf8b-4c84-91ee-52771307fa9c"). InnerVolumeSpecName "kube-api-access-bhglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:01.275916 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.275783 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-xks-ca" pod "dc445bee-cf8b-4c84-91ee-52771307fa9c" (UID: "dc445bee-cf8b-4c84-91ee-52771307fa9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:01.375202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.375116 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:18:01.375202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.375151 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc445bee-cf8b-4c84-91ee-52771307fa9c-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:18:01.375202 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:01.375166 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhglj\" (UniqueName: \"kubernetes.io/projected/dc445bee-cf8b-4c84-91ee-52771307fa9c-kube-api-access-bhglj\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:18:02.114765 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:02.114730 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" Nov 03 23:18:02.114984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:02.114755 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj" event={"ID":"dc445bee-cf8b-4c84-91ee-52771307fa9c","Type":"ContainerDied","Data":"bf124b5eeb9ee7c4b3569ff52eb6d9185dc4de094fc53dfa6e94c6a56baf321f"} Nov 03 23:18:02.114984 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:02.114840 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf124b5eeb9ee7c4b3569ff52eb6d9185dc4de094fc53dfa6e94c6a56baf321f" Nov 03 23:18:12.750715 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:12.750670 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:18:23.151137 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:23.151085 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:18:24.500046 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:18:24.499999 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:18:38.262179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:38.262140 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:18:44.362660 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:44.362622 5272 scope.go:122] "RemoveContainer" containerID="00961b035cab0653308cd999265becc8debd84aecf1dbbaa304412765c0ea759" Nov 03 23:18:45.044637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:45.044599 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z"] Nov 03 23:18:45.091340 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:45.091307 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nph6z"] Nov 03 23:18:45.491448 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:45.491369 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ef3981-0023-4948-9df7-1ee089a9d99e" path="/var/lib/kubelet/pods/16ef3981-0023-4948-9df7-1ee089a9d99e/volumes" Nov 03 23:18:48.240199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.240163 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2"] Nov 03 23:18:48.240626 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.240574 5272 cpu_manager.go:414] "containerMap: removing container" podUID="dc445bee-cf8b-4c84-91ee-52771307fa9c" containerName="revoke-keys" Nov 03 23:18:48.240626 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.240586 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc445bee-cf8b-4c84-91ee-52771307fa9c" containerName="revoke-keys" Nov 03 23:18:48.240753 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.240659 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc445bee-cf8b-4c84-91ee-52771307fa9c" containerName="revoke-keys" Nov 03 23:18:48.245094 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.245072 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.274171 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.274129 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.274366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.274206 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.274366 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.274280 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twf5\" (UniqueName: \"kubernetes.io/projected/01ed66a4-775a-4cfe-bb00-001be20880ee-kube-api-access-8twf5\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.342469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.342430 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:18:48.342469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.342446 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:18:48.342706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.342452 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:18:48.342706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.342516 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:18:48.344426 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.344400 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2"] Nov 03 23:18:48.375526 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.375494 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twf5\" (UniqueName: \"kubernetes.io/projected/01ed66a4-775a-4cfe-bb00-001be20880ee-kube-api-access-8twf5\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.375708 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.375602 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.375708 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.375647 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.376286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.376263 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.378029 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.378007 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.433514 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.433485 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twf5\" (UniqueName: \"kubernetes.io/projected/01ed66a4-775a-4cfe-bb00-001be20880ee-kube-api-access-8twf5\") pod \"maas-api-revoke-sub-e2fc9c5b-kvlj2\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.560586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.560555 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:48.931717 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:48.931627 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2"] Nov 03 23:18:48.938543 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:18:48.938508 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ed66a4_775a_4cfe_bb00_001be20880ee.slice/crio-eefddfa8f217848698eaf055ad3564520a57bd5ab5d5833e0ee5fcaaa879cad1 WatchSource:0}: Error finding container eefddfa8f217848698eaf055ad3564520a57bd5ab5d5833e0ee5fcaaa879cad1: Status 404 returned error can't find the container with id eefddfa8f217848698eaf055ad3564520a57bd5ab5d5833e0ee5fcaaa879cad1 Nov 03 23:18:49.468739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:49.468701 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" event={"ID":"01ed66a4-775a-4cfe-bb00-001be20880ee","Type":"ContainerStarted","Data":"34a1fd89d796507fde9afb8eba1bee57571158348e1ec09929c8c766ecac5434"} Nov 03 23:18:49.469193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:49.468761 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" event={"ID":"01ed66a4-775a-4cfe-bb00-001be20880ee","Type":"ContainerStarted","Data":"eefddfa8f217848698eaf055ad3564520a57bd5ab5d5833e0ee5fcaaa879cad1"} Nov 03 23:18:49.581131 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:49.581065 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" podStartSLOduration=1.581046269 podStartE2EDuration="1.581046269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:18:49.579138125 +0000 UTC m=+3310.201446434" watchObservedRunningTime="2026-11-03 23:18:49.581046269 +0000 UTC m=+3310.203354577" Nov 03 23:18:50.474947 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:50.474913 5272 generic.go:356] "Generic (PLEG): container finished" podID="01ed66a4-775a-4cfe-bb00-001be20880ee" containerID="34a1fd89d796507fde9afb8eba1bee57571158348e1ec09929c8c766ecac5434" exitCode=0 Nov 03 23:18:50.475413 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:50.474997 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" event={"ID":"01ed66a4-775a-4cfe-bb00-001be20880ee","Type":"ContainerDied","Data":"34a1fd89d796507fde9afb8eba1bee57571158348e1ec09929c8c766ecac5434"} Nov 03 23:18:51.615065 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.615038 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:51.706146 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.706109 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-openshift-ca\") pod \"01ed66a4-775a-4cfe-bb00-001be20880ee\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " Nov 03 23:18:51.706316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.706155 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ed66a4-775a-4cfe-bb00-001be20880ee-kube-api-access-8twf5\" (UniqueName: \"kubernetes.io/projected/01ed66a4-775a-4cfe-bb00-001be20880ee-kube-api-access-8twf5\") pod \"01ed66a4-775a-4cfe-bb00-001be20880ee\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " Nov 03 23:18:51.706316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.706187 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-xks-ca\") pod \"01ed66a4-775a-4cfe-bb00-001be20880ee\" (UID: \"01ed66a4-775a-4cfe-bb00-001be20880ee\") " Nov 03 23:18:51.706553 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.706521 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-openshift-ca" pod "01ed66a4-775a-4cfe-bb00-001be20880ee" (UID: "01ed66a4-775a-4cfe-bb00-001be20880ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:51.708381 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.708359 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ed66a4-775a-4cfe-bb00-001be20880ee-kube-api-access-8twf5" pod "01ed66a4-775a-4cfe-bb00-001be20880ee" (UID: "01ed66a4-775a-4cfe-bb00-001be20880ee"). InnerVolumeSpecName "kube-api-access-8twf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:51.708468 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.708358 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-xks-ca" pod "01ed66a4-775a-4cfe-bb00-001be20880ee" (UID: "01ed66a4-775a-4cfe-bb00-001be20880ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:51.806972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.806941 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:18:51.806972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.806969 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twf5\" (UniqueName: \"kubernetes.io/projected/01ed66a4-775a-4cfe-bb00-001be20880ee-kube-api-access-8twf5\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:18:51.807156 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:51.806980 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ed66a4-775a-4cfe-bb00-001be20880ee-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:18:52.487900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:52.487863 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" Nov 03 23:18:52.487900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:52.487887 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2" event={"ID":"01ed66a4-775a-4cfe-bb00-001be20880ee","Type":"ContainerDied","Data":"eefddfa8f217848698eaf055ad3564520a57bd5ab5d5833e0ee5fcaaa879cad1"} Nov 03 23:18:52.488189 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:52.487928 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eefddfa8f217848698eaf055ad3564520a57bd5ab5d5833e0ee5fcaaa879cad1" Nov 03 23:18:54.500024 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:18:54.499987 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:18:59.038239 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.038205 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc"] Nov 03 23:18:59.038625 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.038588 5272 cpu_manager.go:414] "containerMap: removing container" podUID="01ed66a4-775a-4cfe-bb00-001be20880ee" containerName="revoke-keys" Nov 03 23:18:59.038625 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.038599 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ed66a4-775a-4cfe-bb00-001be20880ee" containerName="revoke-keys" Nov 03 23:18:59.038708 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.038659 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ed66a4-775a-4cfe-bb00-001be20880ee" containerName="revoke-keys" Nov 03 23:18:59.041723 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.041700 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.071780 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.071750 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.071950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.071825 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdw8z\" (UniqueName: \"kubernetes.io/projected/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-kube-api-access-kdw8z\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.071950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.071914 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.090542 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.090514 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:18:59.090691 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.090543 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:18:59.137091 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.137061 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:18:59.139030 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.139006 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc"] Nov 03 23:18:59.173081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.173052 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.173263 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.173110 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdw8z\" (UniqueName: \"kubernetes.io/projected/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-kube-api-access-kdw8z\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.173263 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.173136 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.173770 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.173744 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.175423 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.175403 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.191051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.191022 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:18:59.230586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.230564 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdw8z\" (UniqueName: \"kubernetes.io/projected/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-kube-api-access-kdw8z\") pod \"maas-api-revoke-sub-5a29efb3-v6xsc\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.355111 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.355026 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:18:59.725682 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:18:59.725593 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc"] Nov 03 23:18:59.731899 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:18:59.731866 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8feaa8_8ccf_4075_a8cb_45bc25fcbb61.slice/crio-efda9213103859358caea17e53fb881afc2a410f5b387d93b051cc6c15b3b277 WatchSource:0}: Error finding container efda9213103859358caea17e53fb881afc2a410f5b387d93b051cc6c15b3b277: Status 404 returned error can't find the container with id efda9213103859358caea17e53fb881afc2a410f5b387d93b051cc6c15b3b277 Nov 03 23:19:00.529400 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:00.529299 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" event={"ID":"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61","Type":"ContainerStarted","Data":"3db8ab4e34a7ae3603d130515753b8f71b16be5a9c1ac49c3ffd930b311e6811"} Nov 03 23:19:00.529400 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:00.529364 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" event={"ID":"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61","Type":"ContainerStarted","Data":"efda9213103859358caea17e53fb881afc2a410f5b387d93b051cc6c15b3b277"} Nov 03 23:19:00.638859 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:00.638780 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" podStartSLOduration=2.6387658800000002 podStartE2EDuration="2.63876588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:19:00.638109007 +0000 UTC m=+3321.260417315" watchObservedRunningTime="2026-11-03 23:19:00.63876588 +0000 UTC m=+3321.261074197" Nov 03 23:19:01.240319 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:01.240280 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:19:01.535751 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:01.535717 5272 generic.go:356] "Generic (PLEG): container finished" podID="6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" containerID="3db8ab4e34a7ae3603d130515753b8f71b16be5a9c1ac49c3ffd930b311e6811" exitCode=0 Nov 03 23:19:01.536204 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:01.535831 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" event={"ID":"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61","Type":"ContainerDied","Data":"3db8ab4e34a7ae3603d130515753b8f71b16be5a9c1ac49c3ffd930b311e6811"} Nov 03 23:19:03.429190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.428754 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:19:03.534157 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.534115 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-xks-ca\") pod \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " Nov 03 23:19:03.534505 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.534476 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-openshift-ca\") pod \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " Nov 03 23:19:03.534763 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.534742 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-kube-api-access-kdw8z\" (UniqueName: \"kubernetes.io/projected/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-kube-api-access-kdw8z\") pod \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\" (UID: \"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61\") " Nov 03 23:19:03.538970 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.538908 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-kube-api-access-kdw8z" pod "6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" (UID: "6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61"). InnerVolumeSpecName "kube-api-access-kdw8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:19:03.539323 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.539301 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-openshift-ca" pod "6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" (UID: "6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:19:03.542265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.542242 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-xks-ca" pod "6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" (UID: "6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:19:03.586652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.586601 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" event={"ID":"6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61","Type":"ContainerDied","Data":"efda9213103859358caea17e53fb881afc2a410f5b387d93b051cc6c15b3b277"} Nov 03 23:19:03.586951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.586919 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efda9213103859358caea17e53fb881afc2a410f5b387d93b051cc6c15b3b277" Nov 03 23:19:03.587215 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.587168 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc" Nov 03 23:19:03.640589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.640548 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:19:03.640868 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.640847 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:19:03.640995 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:03.640981 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdw8z\" (UniqueName: \"kubernetes.io/projected/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61-kube-api-access-kdw8z\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:19:11.850044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:11.850003 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:19:16.044886 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:16.044853 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd"] Nov 03 23:19:16.091541 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:16.091502 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-52vxd"] Nov 03 23:19:17.491526 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:17.491491 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846c8abe-4881-45fe-bc62-33012b1d1662" path="/var/lib/kubelet/pods/846c8abe-4881-45fe-bc62-33012b1d1662/volumes" Nov 03 23:19:24.500032 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:19:24.499951 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:19:29.055107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:29.055068 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp"] Nov 03 23:19:29.101648 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:29.101614 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-lfdjp"] Nov 03 23:19:29.491133 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:29.491055 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7a7c83-6eb2-4531-a1a7-86aaf371edc5" path="/var/lib/kubelet/pods/5e7a7c83-6eb2-4531-a1a7-86aaf371edc5/volumes" Nov 03 23:19:35.815838 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.815780 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj"] Nov 03 23:19:35.862116 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.862079 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b78pj"] Nov 03 23:19:35.939928 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.939895 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj"] Nov 03 23:19:35.940287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.940274 5272 cpu_manager.go:414] "containerMap: removing container" podUID="6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" containerName="revoke-keys" Nov 03 23:19:35.940344 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.940288 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" containerName="revoke-keys" Nov 03 23:19:35.940384 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.940375 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" containerName="revoke-keys" Nov 03 23:19:35.944819 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.944777 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:35.972664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.972631 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302803cb-b541-499d-87da-444d2d84a88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:35.972859 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.972698 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czj6p\" (UniqueName: \"kubernetes.io/projected/302803cb-b541-499d-87da-444d2d84a88e-kube-api-access-czj6p\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:35.972859 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.972839 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302803cb-b541-499d-87da-444d2d84a88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:35.996014 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.995985 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:19:35.996014 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.996005 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:19:35.996283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.996035 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:19:35.996379 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.996353 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:19:35.998286 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:35.998262 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj"] Nov 03 23:19:36.073724 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.073641 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302803cb-b541-499d-87da-444d2d84a88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:36.073724 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.073701 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czj6p\" (UniqueName: \"kubernetes.io/projected/302803cb-b541-499d-87da-444d2d84a88e-kube-api-access-czj6p\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:36.073942 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.073739 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302803cb-b541-499d-87da-444d2d84a88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:36.074321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.074293 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302803cb-b541-499d-87da-444d2d84a88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:36.076107 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.076088 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302803cb-b541-499d-87da-444d2d84a88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:36.131142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.131117 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czj6p\" (UniqueName: \"kubernetes.io/projected/302803cb-b541-499d-87da-444d2d84a88e-kube-api-access-czj6p\") pod \"maas-api-revoke-sub-3d4156f1-shfkj\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:36.257150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.257117 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:36.661630 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.661584 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj"] Nov 03 23:19:36.668929 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:19:36.668900 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302803cb_b541_499d_87da_444d2d84a88e.slice/crio-021887ecd1115e824bc85e2d93e5c5a290e0987ddfa6648b2f287d0c531722c7 WatchSource:0}: Error finding container 021887ecd1115e824bc85e2d93e5c5a290e0987ddfa6648b2f287d0c531722c7: Status 404 returned error can't find the container with id 021887ecd1115e824bc85e2d93e5c5a290e0987ddfa6648b2f287d0c531722c7 Nov 03 23:19:36.750687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:36.750648 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" event={"ID":"302803cb-b541-499d-87da-444d2d84a88e","Type":"ContainerStarted","Data":"021887ecd1115e824bc85e2d93e5c5a290e0987ddfa6648b2f287d0c531722c7"} Nov 03 23:19:37.491162 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:37.491128 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc445bee-cf8b-4c84-91ee-52771307fa9c" path="/var/lib/kubelet/pods/dc445bee-cf8b-4c84-91ee-52771307fa9c/volumes" Nov 03 23:19:37.757508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:37.757415 5272 generic.go:356] "Generic (PLEG): container finished" podID="302803cb-b541-499d-87da-444d2d84a88e" containerID="e2db33772996d543321242015c7023165087d48993efabe9b4d6c0b8e559fd23" exitCode=0 Nov 03 23:19:37.757670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:37.757500 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" event={"ID":"302803cb-b541-499d-87da-444d2d84a88e","Type":"ContainerDied","Data":"e2db33772996d543321242015c7023165087d48993efabe9b4d6c0b8e559fd23"} Nov 03 23:19:38.894180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:38.894155 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:39.001760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.001726 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302803cb-b541-499d-87da-444d2d84a88e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302803cb-b541-499d-87da-444d2d84a88e-maas-api-xks-ca\") pod \"302803cb-b541-499d-87da-444d2d84a88e\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " Nov 03 23:19:39.001952 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.001864 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302803cb-b541-499d-87da-444d2d84a88e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302803cb-b541-499d-87da-444d2d84a88e-maas-api-openshift-ca\") pod \"302803cb-b541-499d-87da-444d2d84a88e\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " Nov 03 23:19:39.001952 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.001897 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302803cb-b541-499d-87da-444d2d84a88e-kube-api-access-czj6p\" (UniqueName: \"kubernetes.io/projected/302803cb-b541-499d-87da-444d2d84a88e-kube-api-access-czj6p\") pod \"302803cb-b541-499d-87da-444d2d84a88e\" (UID: \"302803cb-b541-499d-87da-444d2d84a88e\") " Nov 03 23:19:39.002210 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.002186 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302803cb-b541-499d-87da-444d2d84a88e-maas-api-openshift-ca" pod "302803cb-b541-499d-87da-444d2d84a88e" (UID: "302803cb-b541-499d-87da-444d2d84a88e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:19:39.004779 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.004740 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302803cb-b541-499d-87da-444d2d84a88e-maas-api-xks-ca" pod "302803cb-b541-499d-87da-444d2d84a88e" (UID: "302803cb-b541-499d-87da-444d2d84a88e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:19:39.005094 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.005068 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302803cb-b541-499d-87da-444d2d84a88e-kube-api-access-czj6p" pod "302803cb-b541-499d-87da-444d2d84a88e" (UID: "302803cb-b541-499d-87da-444d2d84a88e"). InnerVolumeSpecName "kube-api-access-czj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:19:39.103021 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.102989 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302803cb-b541-499d-87da-444d2d84a88e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:19:39.103021 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.103018 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czj6p\" (UniqueName: \"kubernetes.io/projected/302803cb-b541-499d-87da-444d2d84a88e-kube-api-access-czj6p\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:19:39.103223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.103028 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302803cb-b541-499d-87da-444d2d84a88e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:19:39.768314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.768222 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" event={"ID":"302803cb-b541-499d-87da-444d2d84a88e","Type":"ContainerDied","Data":"021887ecd1115e824bc85e2d93e5c5a290e0987ddfa6648b2f287d0c531722c7"} Nov 03 23:19:39.768314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.768275 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="021887ecd1115e824bc85e2d93e5c5a290e0987ddfa6648b2f287d0c531722c7" Nov 03 23:19:39.768314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:39.768233 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj" Nov 03 23:19:44.384182 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:44.384140 5272 scope.go:122] "RemoveContainer" containerID="bca6203b28d6161e2788a99f602e46d1b57c77ccde22800de7bc856fca9ce178" Nov 03 23:19:50.659377 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:19:50.659339 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:19:54.499456 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:19:54.499420 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:20:01.371326 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:01.371286 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:20:16.683095 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.683054 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8"] Nov 03 23:20:16.684325 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.684294 5272 cpu_manager.go:414] "containerMap: removing container" podUID="302803cb-b541-499d-87da-444d2d84a88e" containerName="revoke-keys" Nov 03 23:20:16.684512 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.684494 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302803cb-b541-499d-87da-444d2d84a88e" containerName="revoke-keys" Nov 03 23:20:16.684731 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.684718 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="302803cb-b541-499d-87da-444d2d84a88e" containerName="revoke-keys" Nov 03 23:20:16.689027 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.688996 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:16.743506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.743466 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8"] Nov 03 23:20:16.785544 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.785511 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-aec78cb5-dockercfg-8g9v5\"" Nov 03 23:20:16.785544 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.785517 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1376dc\"" Nov 03 23:20:16.810738 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.810701 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa78de4d-4928-4409-aec5-da978727b939-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:16.810943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.810843 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvr2\" (UniqueName: \"kubernetes.io/projected/fa78de4d-4928-4409-aec5-da978727b939-kube-api-access-2lvr2\") pod \"payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:16.838753 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.838715 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6"] Nov 03 23:20:16.842390 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.842368 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:16.896488 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.896455 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6"] Nov 03 23:20:16.911398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.911367 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvr2\" (UniqueName: \"kubernetes.io/projected/fa78de4d-4928-4409-aec5-da978727b939-kube-api-access-2lvr2\") pod \"payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:16.911579 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.911447 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa78de4d-4928-4409-aec5-da978727b939-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:16.912163 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.912141 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa78de4d-4928-4409-aec5-da978727b939-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:16.968871 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:16.968787 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvr2\" (UniqueName: \"kubernetes.io/projected/fa78de4d-4928-4409-aec5-da978727b939-kube-api-access-2lvr2\") pod \"payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:17.011146 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.011116 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:17.012436 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.012412 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.012500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.012465 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdx9\" (UniqueName: \"kubernetes.io/projected/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-kube-api-access-qvdx9\") pod \"payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.113992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.113949 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdx9\" (UniqueName: \"kubernetes.io/projected/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-kube-api-access-qvdx9\") pod \"payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.114198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.114057 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.114715 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.114689 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.172028 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.171991 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdx9\" (UniqueName: \"kubernetes.io/projected/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-kube-api-access-qvdx9\") pod \"payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.384825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.384766 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8"] Nov 03 23:20:17.391874 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:20:17.391844 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa78de4d_4928_4409_aec5_da978727b939.slice/crio-2e6dee43167a0988d4ccf68690027bbf580f574476219b9d7be0ab3e53ad3c37 WatchSource:0}: Error finding container 2e6dee43167a0988d4ccf68690027bbf580f574476219b9d7be0ab3e53ad3c37: Status 404 returned error can't find the container with id 2e6dee43167a0988d4ccf68690027bbf580f574476219b9d7be0ab3e53ad3c37 Nov 03 23:20:17.454631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.454605 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.822763 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.822719 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6"] Nov 03 23:20:17.830509 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:20:17.830483 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d0dbad_a7a0_4233_a94d_18a742e5aa9c.slice/crio-a2478605b02699dbcb12e3f6fc9276bfa784f715ea3632698412508d41dca95f WatchSource:0}: Error finding container a2478605b02699dbcb12e3f6fc9276bfa784f715ea3632698412508d41dca95f: Status 404 returned error can't find the container with id a2478605b02699dbcb12e3f6fc9276bfa784f715ea3632698412508d41dca95f Nov 03 23:20:17.976067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.976030 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" event={"ID":"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c","Type":"ContainerStarted","Data":"a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca"} Nov 03 23:20:17.976211 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.976101 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" event={"ID":"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c","Type":"ContainerStarted","Data":"a2478605b02699dbcb12e3f6fc9276bfa784f715ea3632698412508d41dca95f"} Nov 03 23:20:17.976211 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.976136 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:17.977814 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.977772 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" event={"ID":"fa78de4d-4928-4409-aec5-da978727b939","Type":"ContainerStarted","Data":"9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445"} Nov 03 23:20:17.977915 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.977853 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:17.977915 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:17.977878 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" event={"ID":"fa78de4d-4928-4409-aec5-da978727b939","Type":"ContainerStarted","Data":"2e6dee43167a0988d4ccf68690027bbf580f574476219b9d7be0ab3e53ad3c37"} Nov 03 23:20:18.087228 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:18.087152 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" podStartSLOduration=2.087137648 podStartE2EDuration="2.087137648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:18.084591225 +0000 UTC m=+3398.706899535" watchObservedRunningTime="2026-11-03 23:20:18.087137648 +0000 UTC m=+3398.709445995" Nov 03 23:20:18.192664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:18.192617 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" podStartSLOduration=2.192603568 podStartE2EDuration="2.192603568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:18.192490267 +0000 UTC m=+3398.814798566" watchObservedRunningTime="2026-11-03 23:20:18.192603568 +0000 UTC m=+3398.814911876" Nov 03 23:20:24.500015 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:20:24.499975 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:20:27.065121 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:27.065083 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:20:28.985777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:28.985740 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:20:28.986215 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:28.985857 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:20:44.410944 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:44.410907 5272 scope.go:122] "RemoveContainer" containerID="88057c8d88559080f012a7041b35d91e7dbed1043021e372c0eea071a32367b8" Nov 03 23:20:44.420074 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:44.420032 5272 scope.go:122] "RemoveContainer" containerID="85aa36967dcf53f940ea228dbcedd3f47bc823cb0c5c76931c82a49fe697e5ce" Nov 03 23:20:45.839722 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:45.839685 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:20:53.044454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:53.044367 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq"] Nov 03 23:20:53.090944 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:53.090909 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5bstq"] Nov 03 23:20:53.491234 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:20:53.491157 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38199f36-7b56-446d-b872-e59e48a1b65c" path="/var/lib/kubelet/pods/38199f36-7b56-446d-b872-e59e48a1b65c/volumes" Nov 03 23:20:54.499241 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:20:54.499207 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:21:21.044906 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:21.044867 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5"] Nov 03 23:21:21.091096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:21.091068 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-829w5"] Nov 03 23:21:21.491995 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:21.491905 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5" path="/var/lib/kubelet/pods/be5a3cf5-fea3-455b-9a0a-ad466bbcc2b5/volumes" Nov 03 23:21:23.949918 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:23.949883 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:21:24.499149 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:21:24.499102 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:21:37.883759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.883721 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6"] Nov 03 23:21:37.887656 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.887639 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:37.936640 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.936617 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:21:37.968027 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.968004 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:37.968164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.968062 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:37.968253 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.968239 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9kj\" (UniqueName: \"kubernetes.io/projected/4246b50d-11d6-4bee-aacf-45164f0022de-kube-api-access-7f9kj\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:37.983410 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.983361 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:21:37.983410 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.983373 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:21:37.985175 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:37.985152 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6"] Nov 03 23:21:38.037155 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.037133 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:21:38.069396 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.069368 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9kj\" (UniqueName: \"kubernetes.io/projected/4246b50d-11d6-4bee-aacf-45164f0022de-kube-api-access-7f9kj\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:38.069501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.069414 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:38.069501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.069473 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:38.070103 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.070074 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:38.071720 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.071701 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:38.126465 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.126434 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9kj\" (UniqueName: \"kubernetes.io/projected/4246b50d-11d6-4bee-aacf-45164f0022de-kube-api-access-7f9kj\") pod \"maas-api-revoke-sub-650e8641-2q9n6\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:38.202195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.202138 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:38.569835 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:38.569793 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6"] Nov 03 23:21:39.399354 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:39.399323 5272 generic.go:356] "Generic (PLEG): container finished" podID="4246b50d-11d6-4bee-aacf-45164f0022de" containerID="980520e3177a9d2a7e9f953daf981267fc297bbf6000bc66468f44174a4fc157" exitCode=0 Nov 03 23:21:39.399705 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:39.399377 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" event={"ID":"4246b50d-11d6-4bee-aacf-45164f0022de","Type":"ContainerDied","Data":"980520e3177a9d2a7e9f953daf981267fc297bbf6000bc66468f44174a4fc157"} Nov 03 23:21:39.399705 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:39.399420 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" event={"ID":"4246b50d-11d6-4bee-aacf-45164f0022de","Type":"ContainerStarted","Data":"0720a19413a01a8d644e66a20eb2eca068575e4b06d84478109f5ccb2bbb2c03"} Nov 03 23:21:40.529996 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.529971 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:40.693989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.693912 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-openshift-ca\") pod \"4246b50d-11d6-4bee-aacf-45164f0022de\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " Nov 03 23:21:40.693989 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.693945 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4246b50d-11d6-4bee-aacf-45164f0022de-kube-api-access-7f9kj\" (UniqueName: \"kubernetes.io/projected/4246b50d-11d6-4bee-aacf-45164f0022de-kube-api-access-7f9kj\") pod \"4246b50d-11d6-4bee-aacf-45164f0022de\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " Nov 03 23:21:40.694203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.694004 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-xks-ca\") pod \"4246b50d-11d6-4bee-aacf-45164f0022de\" (UID: \"4246b50d-11d6-4bee-aacf-45164f0022de\") " Nov 03 23:21:40.694275 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.694239 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-openshift-ca" pod "4246b50d-11d6-4bee-aacf-45164f0022de" (UID: "4246b50d-11d6-4bee-aacf-45164f0022de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:21:40.696068 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.696039 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-xks-ca" pod "4246b50d-11d6-4bee-aacf-45164f0022de" (UID: "4246b50d-11d6-4bee-aacf-45164f0022de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:21:40.696238 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.696218 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4246b50d-11d6-4bee-aacf-45164f0022de-kube-api-access-7f9kj" pod "4246b50d-11d6-4bee-aacf-45164f0022de" (UID: "4246b50d-11d6-4bee-aacf-45164f0022de"). InnerVolumeSpecName "kube-api-access-7f9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:21:40.794972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.794944 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:21:40.794972 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.794967 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4246b50d-11d6-4bee-aacf-45164f0022de-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:21:40.795138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:40.794979 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9kj\" (UniqueName: \"kubernetes.io/projected/4246b50d-11d6-4bee-aacf-45164f0022de-kube-api-access-7f9kj\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:21:41.410283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:41.410242 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" event={"ID":"4246b50d-11d6-4bee-aacf-45164f0022de","Type":"ContainerDied","Data":"0720a19413a01a8d644e66a20eb2eca068575e4b06d84478109f5ccb2bbb2c03"} Nov 03 23:21:41.410464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:41.410295 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0720a19413a01a8d644e66a20eb2eca068575e4b06d84478109f5ccb2bbb2c03" Nov 03 23:21:41.410464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:41.410255 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6" Nov 03 23:21:50.095559 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.095526 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5"] Nov 03 23:21:50.095986 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.095912 5272 cpu_manager.go:414] "containerMap: removing container" podUID="4246b50d-11d6-4bee-aacf-45164f0022de" containerName="revoke-keys" Nov 03 23:21:50.095986 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.095923 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4246b50d-11d6-4bee-aacf-45164f0022de" containerName="revoke-keys" Nov 03 23:21:50.096067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.096005 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="4246b50d-11d6-4bee-aacf-45164f0022de" containerName="revoke-keys" Nov 03 23:21:50.099205 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.099183 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.148014 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.147990 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:21:50.148014 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.147999 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:21:50.189986 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.189947 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.190122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.190002 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpt5p\" (UniqueName: \"kubernetes.io/projected/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-kube-api-access-mpt5p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.190178 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.190144 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.194501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.194486 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:21:50.194583 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.194537 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:21:50.196255 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.196232 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5"] Nov 03 23:21:50.291317 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.291283 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.291571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.291341 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.291571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.291368 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpt5p\" (UniqueName: \"kubernetes.io/projected/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-kube-api-access-mpt5p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.292075 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.292052 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.293651 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.293633 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.349056 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.348999 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpt5p\" (UniqueName: \"kubernetes.io/projected/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-kube-api-access-mpt5p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.411758 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.411731 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:50.782071 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:50.781981 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5"] Nov 03 23:21:50.790318 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:21:50.790275 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f16dd4_a0d0_4dd4_825f_5b15f17c7b11.slice/crio-7cac771dba08e6731e8880c875868163d9b1e321934599dfde8931afffb6ba94 WatchSource:0}: Error finding container 7cac771dba08e6731e8880c875868163d9b1e321934599dfde8931afffb6ba94: Status 404 returned error can't find the container with id 7cac771dba08e6731e8880c875868163d9b1e321934599dfde8931afffb6ba94 Nov 03 23:21:51.460601 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:51.460567 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" event={"ID":"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11","Type":"ContainerStarted","Data":"6e35284936cee85422709489a6d3095993995f4198a59a4c4d27de2b785bf52f"} Nov 03 23:21:51.461044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:51.460619 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" event={"ID":"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11","Type":"ContainerStarted","Data":"7cac771dba08e6731e8880c875868163d9b1e321934599dfde8931afffb6ba94"} Nov 03 23:21:51.569840 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:51.569766 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" podStartSLOduration=1.569752034 podStartE2EDuration="1.569752034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:51.569072352 +0000 UTC m=+3492.191380661" watchObservedRunningTime="2026-11-03 23:21:51.569752034 +0000 UTC m=+3492.192060339" Nov 03 23:21:52.044605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:52.044572 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72"] Nov 03 23:21:52.049340 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:52.049316 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lls72"] Nov 03 23:21:52.466287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:52.466207 5272 generic.go:356] "Generic (PLEG): container finished" podID="c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" containerID="6e35284936cee85422709489a6d3095993995f4198a59a4c4d27de2b785bf52f" exitCode=0 Nov 03 23:21:52.466655 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:52.466291 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" event={"ID":"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11","Type":"ContainerDied","Data":"6e35284936cee85422709489a6d3095993995f4198a59a4c4d27de2b785bf52f"} Nov 03 23:21:52.881314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:52.881279 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:21:53.492025 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.491990 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5626e8e-5fd1-4950-9d15-4ee3cddfdaae" path="/var/lib/kubelet/pods/c5626e8e-5fd1-4950-9d15-4ee3cddfdaae/volumes" Nov 03 23:21:53.607865 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.607840 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:53.722708 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.722673 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-xks-ca\") pod \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " Nov 03 23:21:53.722937 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.722761 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-kube-api-access-mpt5p\" (UniqueName: \"kubernetes.io/projected/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-kube-api-access-mpt5p\") pod \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " Nov 03 23:21:53.722937 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.722839 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-openshift-ca\") pod \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\" (UID: \"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11\") " Nov 03 23:21:53.723471 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.723446 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-openshift-ca" pod "c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" (UID: "c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:21:53.725290 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.725260 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-xks-ca" pod "c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" (UID: "c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:21:53.725469 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.725447 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-kube-api-access-mpt5p" pod "c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" (UID: "c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11"). InnerVolumeSpecName "kube-api-access-mpt5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:21:53.823851 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.823792 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:21:53.823851 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.823844 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpt5p\" (UniqueName: \"kubernetes.io/projected/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-kube-api-access-mpt5p\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:21:53.824016 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:53.823857 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:21:54.485740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:54.485704 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" Nov 03 23:21:54.485740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:54.485722 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5" event={"ID":"c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11","Type":"ContainerDied","Data":"7cac771dba08e6731e8880c875868163d9b1e321934599dfde8931afffb6ba94"} Nov 03 23:21:54.485988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:21:54.485781 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cac771dba08e6731e8880c875868163d9b1e321934599dfde8931afffb6ba94" Nov 03 23:21:54.505215 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:21:54.505185 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:22:01.618299 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:01.618264 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8"] Nov 03 23:22:01.724811 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:01.724766 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6"] Nov 03 23:22:06.618798 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:06.618729 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" podUID="fa78de4d-4928-4409-aec5-da978727b939" containerName="payload-processing" containerID="cri-o://9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445" gracePeriod=25 Nov 03 23:22:06.725882 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:06.725832 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" podUID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" containerName="payload-pre-processing" containerID="cri-o://a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca" gracePeriod=25 Nov 03 23:22:07.634449 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:07.634412 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5"] Nov 03 23:22:07.661103 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:07.661065 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1376dc-rdkd5"] Nov 03 23:22:08.984527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:08.984485 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" podUID="fa78de4d-4928-4409-aec5-da978727b939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:22:08.984527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:08.984501 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" podUID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:22:09.493155 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:09.493124 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" path="/var/lib/kubelet/pods/c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11/volumes" Nov 03 23:22:10.372351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:10.372313 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:22:16.254426 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.254394 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj"] Nov 03 23:22:16.300441 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.300403 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-shfkj"] Nov 03 23:22:16.377038 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.377005 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d"] Nov 03 23:22:16.377391 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.377378 5272 cpu_manager.go:414] "containerMap: removing container" podUID="c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" containerName="revoke-keys" Nov 03 23:22:16.377435 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.377392 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" containerName="revoke-keys" Nov 03 23:22:16.377483 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.377473 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f16dd4-a0d0-4dd4-825f-5b15f17c7b11" containerName="revoke-keys" Nov 03 23:22:16.380727 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.380706 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.429936 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.429906 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:22:16.429936 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.429917 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:22:16.430125 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.429952 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:22:16.430125 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.429971 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:22:16.435187 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.435164 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d"] Nov 03 23:22:16.444142 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.444116 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.444278 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.444152 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvwp\" (UniqueName: \"kubernetes.io/projected/e52e3e69-087b-4ce0-b384-e58b0b0e399c-kube-api-access-sdvwp\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.444278 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.444265 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.545136 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.545106 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.545309 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.545145 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvwp\" (UniqueName: \"kubernetes.io/projected/e52e3e69-087b-4ce0-b384-e58b0b0e399c-kube-api-access-sdvwp\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.545385 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.545351 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.545992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.545968 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.547454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.547428 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.602871 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.602845 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvwp\" (UniqueName: \"kubernetes.io/projected/e52e3e69-087b-4ce0-b384-e58b0b0e399c-kube-api-access-sdvwp\") pod \"maas-api-revoke-sub-3d4156f1-ksv8d\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:16.693055 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:16.693029 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:17.044031 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.044004 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n"] Nov 03 23:22:17.047071 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.047047 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-slw7n"] Nov 03 23:22:17.060877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.060857 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d"] Nov 03 23:22:17.067868 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:22:17.067841 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52e3e69_087b_4ce0_b384_e58b0b0e399c.slice/crio-cc41483fcfce2ac13089d57673d6ab3cae5ac3496afeb30055cbe358c4593f86 WatchSource:0}: Error finding container cc41483fcfce2ac13089d57673d6ab3cae5ac3496afeb30055cbe358c4593f86: Status 404 returned error can't find the container with id cc41483fcfce2ac13089d57673d6ab3cae5ac3496afeb30055cbe358c4593f86 Nov 03 23:22:17.491997 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.491905 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302803cb-b541-499d-87da-444d2d84a88e" path="/var/lib/kubelet/pods/302803cb-b541-499d-87da-444d2d84a88e/volumes" Nov 03 23:22:17.492528 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.492507 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf274a12-36f2-4710-a581-b49c2b32e5d3" path="/var/lib/kubelet/pods/bf274a12-36f2-4710-a581-b49c2b32e5d3/volumes" Nov 03 23:22:17.601943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.601904 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" event={"ID":"e52e3e69-087b-4ce0-b384-e58b0b0e399c","Type":"ContainerStarted","Data":"37fc7c0c7c39d73931dd9051929cbad1cb77dffa60e9f81b75aa92d4aeff5cdf"} Nov 03 23:22:17.602116 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.601962 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" event={"ID":"e52e3e69-087b-4ce0-b384-e58b0b0e399c","Type":"ContainerStarted","Data":"cc41483fcfce2ac13089d57673d6ab3cae5ac3496afeb30055cbe358c4593f86"} Nov 03 23:22:17.710791 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:17.710740 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" podStartSLOduration=1.710727246 podStartE2EDuration="1.710727246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:22:17.710352578 +0000 UTC m=+3518.332660886" watchObservedRunningTime="2026-11-03 23:22:17.710727246 +0000 UTC m=+3518.333035553" Nov 03 23:22:18.608057 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:18.608020 5272 generic.go:356] "Generic (PLEG): container finished" podID="e52e3e69-087b-4ce0-b384-e58b0b0e399c" containerID="37fc7c0c7c39d73931dd9051929cbad1cb77dffa60e9f81b75aa92d4aeff5cdf" exitCode=0 Nov 03 23:22:18.608454 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:18.608100 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" event={"ID":"e52e3e69-087b-4ce0-b384-e58b0b0e399c","Type":"ContainerDied","Data":"37fc7c0c7c39d73931dd9051929cbad1cb77dffa60e9f81b75aa92d4aeff5cdf"} Nov 03 23:22:18.984907 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:18.984799 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" podUID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:22:18.985073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:18.984832 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" podUID="fa78de4d-4928-4409-aec5-da978727b939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:22:19.738929 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.738904 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:19.875439 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.875339 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-openshift-ca\") pod \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " Nov 03 23:22:19.875439 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.875400 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-xks-ca\") pod \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " Nov 03 23:22:19.875673 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.875468 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52e3e69-087b-4ce0-b384-e58b0b0e399c-kube-api-access-sdvwp\" (UniqueName: \"kubernetes.io/projected/e52e3e69-087b-4ce0-b384-e58b0b0e399c-kube-api-access-sdvwp\") pod \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\" (UID: \"e52e3e69-087b-4ce0-b384-e58b0b0e399c\") " Nov 03 23:22:19.875742 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.875688 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-openshift-ca" pod "e52e3e69-087b-4ce0-b384-e58b0b0e399c" (UID: "e52e3e69-087b-4ce0-b384-e58b0b0e399c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:19.875872 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.875849 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:19.877438 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.877418 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-xks-ca" pod "e52e3e69-087b-4ce0-b384-e58b0b0e399c" (UID: "e52e3e69-087b-4ce0-b384-e58b0b0e399c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:22:19.877554 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.877525 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52e3e69-087b-4ce0-b384-e58b0b0e399c-kube-api-access-sdvwp" pod "e52e3e69-087b-4ce0-b384-e58b0b0e399c" (UID: "e52e3e69-087b-4ce0-b384-e58b0b0e399c"). InnerVolumeSpecName "kube-api-access-sdvwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:19.977341 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.977308 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52e3e69-087b-4ce0-b384-e58b0b0e399c-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:19.977341 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:19.977342 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvwp\" (UniqueName: \"kubernetes.io/projected/e52e3e69-087b-4ce0-b384-e58b0b0e399c-kube-api-access-sdvwp\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:20.619404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:20.619360 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" event={"ID":"e52e3e69-087b-4ce0-b384-e58b0b0e399c","Type":"ContainerDied","Data":"cc41483fcfce2ac13089d57673d6ab3cae5ac3496afeb30055cbe358c4593f86"} Nov 03 23:22:20.619591 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:20.619417 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc41483fcfce2ac13089d57673d6ab3cae5ac3496afeb30055cbe358c4593f86" Nov 03 23:22:20.619591 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:20.619385 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d" Nov 03 23:22:24.501331 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:22:24.501296 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:22:26.870409 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:26.870378 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:22:26.939969 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:26.939934 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa78de4d-4928-4409-aec5-da978727b939-kube-api-access-2lvr2\" (UniqueName: \"kubernetes.io/projected/fa78de4d-4928-4409-aec5-da978727b939-kube-api-access-2lvr2\") pod \"fa78de4d-4928-4409-aec5-da978727b939\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " Nov 03 23:22:26.940132 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:26.940041 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa78de4d-4928-4409-aec5-da978727b939-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa78de4d-4928-4409-aec5-da978727b939-plugins-config-volume\") pod \"fa78de4d-4928-4409-aec5-da978727b939\" (UID: \"fa78de4d-4928-4409-aec5-da978727b939\") " Nov 03 23:22:26.940475 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:26.940440 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa78de4d-4928-4409-aec5-da978727b939-plugins-config-volume" pod "fa78de4d-4928-4409-aec5-da978727b939" (UID: "fa78de4d-4928-4409-aec5-da978727b939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:26.942500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:26.942472 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa78de4d-4928-4409-aec5-da978727b939-kube-api-access-2lvr2" pod "fa78de4d-4928-4409-aec5-da978727b939" (UID: "fa78de4d-4928-4409-aec5-da978727b939"). InnerVolumeSpecName "kube-api-access-2lvr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:26.972223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:26.972199 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:22:27.040814 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.040768 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-plugins-config-volume\") pod \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " Nov 03 23:22:27.041007 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.040988 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-kube-api-access-qvdx9\" (UniqueName: \"kubernetes.io/projected/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-kube-api-access-qvdx9\") pod \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\" (UID: \"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c\") " Nov 03 23:22:27.041209 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.041189 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-plugins-config-volume" pod "d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" (UID: "d7d0dbad-a7a0-4233-a94d-18a742e5aa9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:27.041291 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.041273 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa78de4d-4928-4409-aec5-da978727b939-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:27.041333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.041295 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvr2\" (UniqueName: \"kubernetes.io/projected/fa78de4d-4928-4409-aec5-da978727b939-kube-api-access-2lvr2\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:27.041333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.041304 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:27.043180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.043156 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-kube-api-access-qvdx9" pod "d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" (UID: "d7d0dbad-a7a0-4233-a94d-18a742e5aa9c"). InnerVolumeSpecName "kube-api-access-qvdx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:27.142337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.142307 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvdx9\" (UniqueName: \"kubernetes.io/projected/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c-kube-api-access-qvdx9\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:22:27.655108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.655075 5272 generic.go:356] "Generic (PLEG): container finished" podID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" containerID="a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca" exitCode=0 Nov 03 23:22:27.655275 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.655142 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" Nov 03 23:22:27.655275 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.655157 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" event={"ID":"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c","Type":"ContainerDied","Data":"a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca"} Nov 03 23:22:27.655275 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.655210 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6" event={"ID":"d7d0dbad-a7a0-4233-a94d-18a742e5aa9c","Type":"ContainerDied","Data":"a2478605b02699dbcb12e3f6fc9276bfa784f715ea3632698412508d41dca95f"} Nov 03 23:22:27.655275 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.655243 5272 scope.go:122] "RemoveContainer" containerID="a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca" Nov 03 23:22:27.656648 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.656623 5272 generic.go:356] "Generic (PLEG): container finished" podID="fa78de4d-4928-4409-aec5-da978727b939" containerID="9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445" exitCode=0 Nov 03 23:22:27.656744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.656683 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" Nov 03 23:22:27.656744 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.656716 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" event={"ID":"fa78de4d-4928-4409-aec5-da978727b939","Type":"ContainerDied","Data":"9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445"} Nov 03 23:22:27.656864 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.656772 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8" event={"ID":"fa78de4d-4928-4409-aec5-da978727b939","Type":"ContainerDied","Data":"2e6dee43167a0988d4ccf68690027bbf580f574476219b9d7be0ab3e53ad3c37"} Nov 03 23:22:27.664685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.664670 5272 scope.go:122] "RemoveContainer" containerID="a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca" Nov 03 23:22:27.664995 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:22:27.664971 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca\": container with ID starting with a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca not found: ID does not exist" containerID="a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca" Nov 03 23:22:27.665066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.665002 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca"} err="failed to get container status \"a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca\": rpc error: code = NotFound desc = could not find container \"a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca\": container with ID starting with a1d50cb71f5db238748169a4e6cc4a158a66e10551bcf7f0569a0f6e97daa8ca not found: ID does not exist" Nov 03 23:22:27.665066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.665018 5272 scope.go:122] "RemoveContainer" containerID="9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445" Nov 03 23:22:27.674827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.674811 5272 scope.go:122] "RemoveContainer" containerID="9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445" Nov 03 23:22:27.675092 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:22:27.675073 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445\": container with ID starting with 9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445 not found: ID does not exist" containerID="9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445" Nov 03 23:22:27.675139 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.675100 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445"} err="failed to get container status \"9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445\": rpc error: code = NotFound desc = could not find container \"9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445\": container with ID starting with 9df5ae87c3b935ee1204cbf389b9d1854d09befbf2f463e081311c1418071445 not found: ID does not exist" Nov 03 23:22:27.811269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.811238 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8"] Nov 03 23:22:27.816830 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.816792 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1376dc-6f575447fb7tb8"] Nov 03 23:22:27.916985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.916952 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6"] Nov 03 23:22:27.921249 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:27.921220 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1376dc-5c8ddlnpd6"] Nov 03 23:22:29.491314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:29.491279 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" path="/var/lib/kubelet/pods/d7d0dbad-a7a0-4233-a94d-18a742e5aa9c/volumes" Nov 03 23:22:29.491724 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:29.491681 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa78de4d-4928-4409-aec5-da978727b939" path="/var/lib/kubelet/pods/fa78de4d-4928-4409-aec5-da978727b939/volumes" Nov 03 23:22:44.452341 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:44.452303 5272 scope.go:122] "RemoveContainer" containerID="49741dbd274d6e70156f0383ca44cf27972c6978fa2f9d8eec13ab753c5e1afc" Nov 03 23:22:44.461103 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:22:44.461079 5272 scope.go:122] "RemoveContainer" containerID="6869d862b0c3f47daa2f4c3888cdfd48ab7521b5baef9a09c762a9bc31c76e0d" Nov 03 23:22:54.499456 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:22:54.499421 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:23:24.500206 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:23:24.500167 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:23:44.492022 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:23:44.491987 5272 scope.go:122] "RemoveContainer" containerID="f8fe0acead4ed2a1dcaede3921a3dbf3871b5c7240938d071840650cb07f3e0c" Nov 03 23:23:44.500764 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:23:44.500735 5272 scope.go:122] "RemoveContainer" containerID="acd6ebcaf7e23732a7e79e56334fa3bc3761ccf36aca1788d3fc9cd2b013b429" Nov 03 23:23:44.508678 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:23:44.508653 5272 scope.go:122] "RemoveContainer" containerID="e2497afa6f9fa1d26d961010b037dc1f178435a67c4c853340b5d142d42dff6e" Nov 03 23:23:52.043888 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:23:52.043790 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2"] Nov 03 23:23:52.090085 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:23:52.090053 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kvlj2"] Nov 03 23:23:53.490917 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:23:53.490871 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ed66a4-775a-4cfe-bb00-001be20880ee" path="/var/lib/kubelet/pods/01ed66a4-775a-4cfe-bb00-001be20880ee/volumes" Nov 03 23:23:54.501036 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:23:54.500996 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:24:03.044226 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:03.044189 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc"] Nov 03 23:24:03.090548 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:03.090506 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v6xsc"] Nov 03 23:24:03.491512 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:03.491431 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61" path="/var/lib/kubelet/pods/6b8feaa8-8ccf-4075-a8cb-45bc25fcbb61/volumes" Nov 03 23:24:05.285080 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:05.285047 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:24:09.982328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982290 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z"] Nov 03 23:24:09.982721 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982709 5272 cpu_manager.go:414] "containerMap: removing container" podUID="fa78de4d-4928-4409-aec5-da978727b939" containerName="payload-processing" Nov 03 23:24:09.982770 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982721 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa78de4d-4928-4409-aec5-da978727b939" containerName="payload-processing" Nov 03 23:24:09.982770 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982737 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e52e3e69-087b-4ce0-b384-e58b0b0e399c" containerName="revoke-keys" Nov 03 23:24:09.982770 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982742 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52e3e69-087b-4ce0-b384-e58b0b0e399c" containerName="revoke-keys" Nov 03 23:24:09.982770 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982767 5272 cpu_manager.go:414] "containerMap: removing container" podUID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" containerName="payload-pre-processing" Nov 03 23:24:09.982935 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982776 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" containerName="payload-pre-processing" Nov 03 23:24:09.982935 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982870 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa78de4d-4928-4409-aec5-da978727b939" containerName="payload-processing" Nov 03 23:24:09.982935 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982879 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52e3e69-087b-4ce0-b384-e58b0b0e399c" containerName="revoke-keys" Nov 03 23:24:09.982935 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.982889 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d0dbad-a7a0-4233-a94d-18a742e5aa9c" containerName="payload-pre-processing" Nov 03 23:24:09.987456 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:09.987433 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.036564 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.036533 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:24:10.065144 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.065104 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.065144 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.065147 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klcth\" (UniqueName: \"kubernetes.io/projected/61d83b70-c34e-449e-9795-7324e03dff06-kube-api-access-klcth\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.065352 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.065305 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.082970 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.082933 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:24:10.082970 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.082933 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:24:10.083168 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.082950 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:24:10.084688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.084666 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z"] Nov 03 23:24:10.166035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.165992 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.166035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.166039 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klcth\" (UniqueName: \"kubernetes.io/projected/61d83b70-c34e-449e-9795-7324e03dff06-kube-api-access-klcth\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.166273 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.166118 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.166642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.166620 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.168495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.168472 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.223844 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.223753 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klcth\" (UniqueName: \"kubernetes.io/projected/61d83b70-c34e-449e-9795-7324e03dff06-kube-api-access-klcth\") pod \"maas-api-revoke-sub-d576e68b-vzf7z\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.300745 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.300707 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:10.670268 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:10.670231 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z"] Nov 03 23:24:10.677568 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:24:10.677533 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d83b70_c34e_449e_9795_7324e03dff06.slice/crio-ce72f1ca99f71ca9d3581b75bfabc4594bd230d14d1e3b317fb60bbab4817275 WatchSource:0}: Error finding container ce72f1ca99f71ca9d3581b75bfabc4594bd230d14d1e3b317fb60bbab4817275: Status 404 returned error can't find the container with id ce72f1ca99f71ca9d3581b75bfabc4594bd230d14d1e3b317fb60bbab4817275 Nov 03 23:24:11.191444 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:11.191406 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" event={"ID":"61d83b70-c34e-449e-9795-7324e03dff06","Type":"ContainerStarted","Data":"dbdf2b010af10ecbf6f0a51c002f0d3ba423b71ecb62cb06cea7faff2e4d811a"} Nov 03 23:24:11.191891 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:11.191470 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" event={"ID":"61d83b70-c34e-449e-9795-7324e03dff06","Type":"ContainerStarted","Data":"ce72f1ca99f71ca9d3581b75bfabc4594bd230d14d1e3b317fb60bbab4817275"} Nov 03 23:24:11.299910 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:11.299856 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" podStartSLOduration=2.299837449 podStartE2EDuration="2.299837449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:24:11.299769262 +0000 UTC m=+3631.922077570" watchObservedRunningTime="2026-11-03 23:24:11.299837449 +0000 UTC m=+3631.922145758" Nov 03 23:24:12.197876 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:12.197840 5272 generic.go:356] "Generic (PLEG): container finished" podID="61d83b70-c34e-449e-9795-7324e03dff06" containerID="dbdf2b010af10ecbf6f0a51c002f0d3ba423b71ecb62cb06cea7faff2e4d811a" exitCode=0 Nov 03 23:24:12.198314 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:12.197905 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" event={"ID":"61d83b70-c34e-449e-9795-7324e03dff06","Type":"ContainerDied","Data":"dbdf2b010af10ecbf6f0a51c002f0d3ba423b71ecb62cb06cea7faff2e4d811a"} Nov 03 23:24:13.346478 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.346451 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:13.396531 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.396499 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-openshift-ca\") pod \"61d83b70-c34e-449e-9795-7324e03dff06\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " Nov 03 23:24:13.396695 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.396539 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d83b70-c34e-449e-9795-7324e03dff06-kube-api-access-klcth\" (UniqueName: \"kubernetes.io/projected/61d83b70-c34e-449e-9795-7324e03dff06-kube-api-access-klcth\") pod \"61d83b70-c34e-449e-9795-7324e03dff06\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " Nov 03 23:24:13.396695 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.396587 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-xks-ca\") pod \"61d83b70-c34e-449e-9795-7324e03dff06\" (UID: \"61d83b70-c34e-449e-9795-7324e03dff06\") " Nov 03 23:24:13.396902 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.396879 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-openshift-ca" pod "61d83b70-c34e-449e-9795-7324e03dff06" (UID: "61d83b70-c34e-449e-9795-7324e03dff06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:24:13.398821 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.398774 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-xks-ca" pod "61d83b70-c34e-449e-9795-7324e03dff06" (UID: "61d83b70-c34e-449e-9795-7324e03dff06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:24:13.398931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.398830 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d83b70-c34e-449e-9795-7324e03dff06-kube-api-access-klcth" pod "61d83b70-c34e-449e-9795-7324e03dff06" (UID: "61d83b70-c34e-449e-9795-7324e03dff06"). InnerVolumeSpecName "kube-api-access-klcth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:24:13.498053 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.497966 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klcth\" (UniqueName: \"kubernetes.io/projected/61d83b70-c34e-449e-9795-7324e03dff06-kube-api-access-klcth\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:24:13.498053 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.498006 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:24:13.498053 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:13.498024 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d83b70-c34e-449e-9795-7324e03dff06-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:24:14.212672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:14.212634 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" event={"ID":"61d83b70-c34e-449e-9795-7324e03dff06","Type":"ContainerDied","Data":"ce72f1ca99f71ca9d3581b75bfabc4594bd230d14d1e3b317fb60bbab4817275"} Nov 03 23:24:14.212672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:14.212663 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z" Nov 03 23:24:14.212922 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:14.212688 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce72f1ca99f71ca9d3581b75bfabc4594bd230d14d1e3b317fb60bbab4817275" Nov 03 23:24:22.782835 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:22.782769 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:24:24.500141 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:24:24.500096 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:24:30.216832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:30.203882 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wkwkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:24:30.216832 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:30.203984 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wkwkn" podUID="1461ab8f-e9cf-409b-ad59-2bc90b14463f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:24:32.559600 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:32.559559 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:24:42.009393 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.009360 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947"] Nov 03 23:24:42.009793 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.009759 5272 cpu_manager.go:414] "containerMap: removing container" podUID="61d83b70-c34e-449e-9795-7324e03dff06" containerName="revoke-keys" Nov 03 23:24:42.009793 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.009773 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d83b70-c34e-449e-9795-7324e03dff06" containerName="revoke-keys" Nov 03 23:24:42.009905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.009883 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d83b70-c34e-449e-9795-7324e03dff06" containerName="revoke-keys" Nov 03 23:24:42.014131 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.014099 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.098652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.098608 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.098872 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.098764 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.098940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.098916 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcf8z\" (UniqueName: \"kubernetes.io/projected/b37c5062-c189-4908-98cf-a5ab4c1f0b60-kube-api-access-qcf8z\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.111743 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.111721 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:24:42.111909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.111749 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:24:42.111909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.111774 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:24:42.111909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.111886 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:24:42.113591 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.113570 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947"] Nov 03 23:24:42.199948 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.199913 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf8z\" (UniqueName: \"kubernetes.io/projected/b37c5062-c189-4908-98cf-a5ab4c1f0b60-kube-api-access-qcf8z\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.200164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.199989 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.200164 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.200075 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.200683 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.200650 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.202500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.202474 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.257274 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.257247 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf8z\" (UniqueName: \"kubernetes.io/projected/b37c5062-c189-4908-98cf-a5ab4c1f0b60-kube-api-access-qcf8z\") pod \"maas-api-revoke-sub-d2bc93bc-ct947\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.326424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.326393 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:42.696515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:42.696423 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947"] Nov 03 23:24:42.703691 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:24:42.703658 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37c5062_c189_4908_98cf_a5ab4c1f0b60.slice/crio-208b29309a68c9b16cfad7b65af7affe188083bb2349dfedefb13deaffed51c1 WatchSource:0}: Error finding container 208b29309a68c9b16cfad7b65af7affe188083bb2349dfedefb13deaffed51c1: Status 404 returned error can't find the container with id 208b29309a68c9b16cfad7b65af7affe188083bb2349dfedefb13deaffed51c1 Nov 03 23:24:43.389019 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:43.388978 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" event={"ID":"b37c5062-c189-4908-98cf-a5ab4c1f0b60","Type":"ContainerStarted","Data":"aa0d4cf890f50adf29da1fbfa9f5ebc6e554b231ec27ae11558e85d623c27938"} Nov 03 23:24:43.389439 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:43.389045 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" event={"ID":"b37c5062-c189-4908-98cf-a5ab4c1f0b60","Type":"ContainerStarted","Data":"208b29309a68c9b16cfad7b65af7affe188083bb2349dfedefb13deaffed51c1"} Nov 03 23:24:43.498538 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:43.498486 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" podStartSLOduration=2.498473308 podStartE2EDuration="2.498473308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:24:43.497053486 +0000 UTC m=+3664.119361795" watchObservedRunningTime="2026-11-03 23:24:43.498473308 +0000 UTC m=+3664.120781637" Nov 03 23:24:44.395878 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:44.395836 5272 generic.go:356] "Generic (PLEG): container finished" podID="b37c5062-c189-4908-98cf-a5ab4c1f0b60" containerID="aa0d4cf890f50adf29da1fbfa9f5ebc6e554b231ec27ae11558e85d623c27938" exitCode=0 Nov 03 23:24:44.396360 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:44.395916 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" event={"ID":"b37c5062-c189-4908-98cf-a5ab4c1f0b60","Type":"ContainerDied","Data":"aa0d4cf890f50adf29da1fbfa9f5ebc6e554b231ec27ae11558e85d623c27938"} Nov 03 23:24:44.535644 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:44.535609 5272 scope.go:122] "RemoveContainer" containerID="183c549dbd52d60213a40fcd4241818e7782dce78c5e81e1ba511bdea662ad20" Nov 03 23:24:45.536484 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.536456 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:45.631026 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.630994 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-xks-ca\") pod \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " Nov 03 23:24:45.631246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.631043 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37c5062-c189-4908-98cf-a5ab4c1f0b60-kube-api-access-qcf8z\" (UniqueName: \"kubernetes.io/projected/b37c5062-c189-4908-98cf-a5ab4c1f0b60-kube-api-access-qcf8z\") pod \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " Nov 03 23:24:45.631246 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.631138 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-openshift-ca\") pod \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\" (UID: \"b37c5062-c189-4908-98cf-a5ab4c1f0b60\") " Nov 03 23:24:45.631526 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.631501 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-openshift-ca" pod "b37c5062-c189-4908-98cf-a5ab4c1f0b60" (UID: "b37c5062-c189-4908-98cf-a5ab4c1f0b60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:24:45.633196 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.633170 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-xks-ca" pod "b37c5062-c189-4908-98cf-a5ab4c1f0b60" (UID: "b37c5062-c189-4908-98cf-a5ab4c1f0b60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:24:45.633328 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.633273 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37c5062-c189-4908-98cf-a5ab4c1f0b60-kube-api-access-qcf8z" pod "b37c5062-c189-4908-98cf-a5ab4c1f0b60" (UID: "b37c5062-c189-4908-98cf-a5ab4c1f0b60"). InnerVolumeSpecName "kube-api-access-qcf8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:24:45.732320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.732239 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:24:45.732320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.732271 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c5062-c189-4908-98cf-a5ab4c1f0b60-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:24:45.732320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:45.732281 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcf8z\" (UniqueName: \"kubernetes.io/projected/b37c5062-c189-4908-98cf-a5ab4c1f0b60-kube-api-access-qcf8z\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:24:46.405631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:46.405589 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" event={"ID":"b37c5062-c189-4908-98cf-a5ab4c1f0b60","Type":"ContainerDied","Data":"208b29309a68c9b16cfad7b65af7affe188083bb2349dfedefb13deaffed51c1"} Nov 03 23:24:46.405631 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:46.405619 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947" Nov 03 23:24:46.405882 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:46.405654 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208b29309a68c9b16cfad7b65af7affe188083bb2349dfedefb13deaffed51c1" Nov 03 23:24:54.499890 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:24:54.499854 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:24:54.877711 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:24:54.877678 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:25:12.182879 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.180553 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5"] Nov 03 23:25:12.185011 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.184981 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b37c5062-c189-4908-98cf-a5ab4c1f0b60" containerName="revoke-keys" Nov 03 23:25:12.185210 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.185194 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37c5062-c189-4908-98cf-a5ab4c1f0b60" containerName="revoke-keys" Nov 03 23:25:12.185497 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.185483 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37c5062-c189-4908-98cf-a5ab4c1f0b60" containerName="revoke-keys" Nov 03 23:25:12.205518 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.205489 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.241684 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.241653 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5"] Nov 03 23:25:12.285668 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.285630 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cbd086a-e80b-4e23-92fc-3aca371b5df2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.285866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.285694 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwzr\" (UniqueName: \"kubernetes.io/projected/3cbd086a-e80b-4e23-92fc-3aca371b5df2-kube-api-access-gtwzr\") pod \"payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.285866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.285703 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e433fd\"" Nov 03 23:25:12.285985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.285875 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d3b21880-dockercfg-fnc9g\"" Nov 03 23:25:12.354986 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.354949 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd"] Nov 03 23:25:12.358535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.358517 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.386767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.386741 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cbd086a-e80b-4e23-92fc-3aca371b5df2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.386922 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.386782 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwzr\" (UniqueName: \"kubernetes.io/projected/3cbd086a-e80b-4e23-92fc-3aca371b5df2-kube-api-access-gtwzr\") pod \"payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.387417 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.387399 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cbd086a-e80b-4e23-92fc-3aca371b5df2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.413036 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.413011 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd"] Nov 03 23:25:12.444682 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.444623 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwzr\" (UniqueName: \"kubernetes.io/projected/3cbd086a-e80b-4e23-92fc-3aca371b5df2-kube-api-access-gtwzr\") pod \"payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.488191 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.488164 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnl4z\" (UniqueName: \"kubernetes.io/projected/60eb8716-e74a-40ca-8330-d8d514a83294-kube-api-access-qnl4z\") pod \"payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.488336 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.488285 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eb8716-e74a-40ca-8330-d8d514a83294-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.526145 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.526119 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:12.589649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.589605 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eb8716-e74a-40ca-8330-d8d514a83294-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.589793 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.589723 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnl4z\" (UniqueName: \"kubernetes.io/projected/60eb8716-e74a-40ca-8330-d8d514a83294-kube-api-access-qnl4z\") pod \"payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.591359 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.591159 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eb8716-e74a-40ca-8330-d8d514a83294-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.647641 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.647607 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnl4z\" (UniqueName: \"kubernetes.io/projected/60eb8716-e74a-40ca-8330-d8d514a83294-kube-api-access-qnl4z\") pod \"payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.671043 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.671019 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:12.894687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:12.894656 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5"] Nov 03 23:25:12.901658 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:25:12.901627 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbd086a_e80b_4e23_92fc_3aca371b5df2.slice/crio-c1f32a230b1071cf007c4b45c52d2c6cddd78f6967eb8212313641fab255d1bb WatchSource:0}: Error finding container c1f32a230b1071cf007c4b45c52d2c6cddd78f6967eb8212313641fab255d1bb: Status 404 returned error can't find the container with id c1f32a230b1071cf007c4b45c52d2c6cddd78f6967eb8212313641fab255d1bb Nov 03 23:25:13.040498 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.040471 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd"] Nov 03 23:25:13.047600 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:25:13.047575 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60eb8716_e74a_40ca_8330_d8d514a83294.slice/crio-dc8b919e7221099cf9a9f9fb8994fde628edf0608583084bdbb7723aeb2441c1 WatchSource:0}: Error finding container dc8b919e7221099cf9a9f9fb8994fde628edf0608583084bdbb7723aeb2441c1: Status 404 returned error can't find the container with id dc8b919e7221099cf9a9f9fb8994fde628edf0608583084bdbb7723aeb2441c1 Nov 03 23:25:13.534680 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.534642 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" event={"ID":"3cbd086a-e80b-4e23-92fc-3aca371b5df2","Type":"ContainerStarted","Data":"48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba"} Nov 03 23:25:13.535178 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.534706 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" event={"ID":"3cbd086a-e80b-4e23-92fc-3aca371b5df2","Type":"ContainerStarted","Data":"c1f32a230b1071cf007c4b45c52d2c6cddd78f6967eb8212313641fab255d1bb"} Nov 03 23:25:13.535178 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.534731 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:13.536198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.536172 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" event={"ID":"60eb8716-e74a-40ca-8330-d8d514a83294","Type":"ContainerStarted","Data":"51b778ba5b19c2fa659ec4491eabcaa8d5fc37135fb0ab1b6270b9af6f041736"} Nov 03 23:25:13.536329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.536218 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" event={"ID":"60eb8716-e74a-40ca-8330-d8d514a83294","Type":"ContainerStarted","Data":"dc8b919e7221099cf9a9f9fb8994fde628edf0608583084bdbb7723aeb2441c1"} Nov 03 23:25:13.536329 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.536262 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:13.644207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.644157 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" podStartSLOduration=1.644143361 podStartE2EDuration="1.644143361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:13.643172083 +0000 UTC m=+3694.265480393" watchObservedRunningTime="2026-11-03 23:25:13.644143361 +0000 UTC m=+3694.266451668" Nov 03 23:25:13.751617 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:13.751554 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" podStartSLOduration=1.751540952 podStartE2EDuration="1.751540952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:13.751533927 +0000 UTC m=+3694.373842235" watchObservedRunningTime="2026-11-03 23:25:13.751540952 +0000 UTC m=+3694.373849276" Nov 03 23:25:22.873417 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:22.873324 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:25:24.500127 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:25:24.500090 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:25:24.544135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:24.544103 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:25:24.544318 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:24.544187 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:25:41.854392 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:41.854347 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:25:44.561385 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:44.561352 5272 scope.go:122] "RemoveContainer" containerID="34a1fd89d796507fde9afb8eba1bee57571158348e1ec09929c8c766ecac5434" Nov 03 23:25:44.569969 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:44.569945 5272 scope.go:122] "RemoveContainer" containerID="e2db33772996d543321242015c7023165087d48993efabe9b4d6c0b8e559fd23" Nov 03 23:25:44.577597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:25:44.577577 5272 scope.go:122] "RemoveContainer" containerID="3db8ab4e34a7ae3603d130515753b8f71b16be5a9c1ac49c3ffd930b311e6811" Nov 03 23:25:54.500703 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:25:54.500665 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:26:01.861331 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:01.861271 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:26:16.679138 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.679100 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct"] Nov 03 23:26:16.682670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.682648 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.732020 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.731986 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:26:16.778495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.778456 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:26:16.778495 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.778460 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:26:16.778712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.778506 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:26:16.780304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.780276 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct"] Nov 03 23:26:16.808198 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.808166 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf529\" (UniqueName: \"kubernetes.io/projected/36dceb03-a50d-40cc-96d3-f939896bdf06-kube-api-access-xf529\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.808361 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.808212 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.808361 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.808234 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.909794 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.909757 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf529\" (UniqueName: \"kubernetes.io/projected/36dceb03-a50d-40cc-96d3-f939896bdf06-kube-api-access-xf529\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.910004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.909828 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.910004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.909850 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.910503 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.910479 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.912413 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.912387 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.967387 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.967308 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf529\" (UniqueName: \"kubernetes.io/projected/36dceb03-a50d-40cc-96d3-f939896bdf06-kube-api-access-xf529\") pod \"maas-api-revoke-sub-2b1a27b4-jbtct\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:16.994912 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:16.994878 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:17.366466 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:17.366428 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct"] Nov 03 23:26:17.374011 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:26:17.373979 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dceb03_a50d_40cc_96d3_f939896bdf06.slice/crio-98254d52ff9a7d0bfa4c724279edbe7d002916433c3a71ba56a03f55d9ff500e WatchSource:0}: Error finding container 98254d52ff9a7d0bfa4c724279edbe7d002916433c3a71ba56a03f55d9ff500e: Status 404 returned error can't find the container with id 98254d52ff9a7d0bfa4c724279edbe7d002916433c3a71ba56a03f55d9ff500e Nov 03 23:26:17.858665 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:17.858626 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" event={"ID":"36dceb03-a50d-40cc-96d3-f939896bdf06","Type":"ContainerStarted","Data":"659011caf716165ef8c876b7d5fa2dfe60d04d4c8d32716de9729643c39d18e1"} Nov 03 23:26:17.859065 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:17.858687 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" event={"ID":"36dceb03-a50d-40cc-96d3-f939896bdf06","Type":"ContainerStarted","Data":"98254d52ff9a7d0bfa4c724279edbe7d002916433c3a71ba56a03f55d9ff500e"} Nov 03 23:26:17.968487 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:17.968378 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" podStartSLOduration=1.968354106 podStartE2EDuration="1.968354106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:17.966731107 +0000 UTC m=+3758.589039416" watchObservedRunningTime="2026-11-03 23:26:17.968354106 +0000 UTC m=+3758.590662448" Nov 03 23:26:18.864679 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:18.864640 5272 generic.go:356] "Generic (PLEG): container finished" podID="36dceb03-a50d-40cc-96d3-f939896bdf06" containerID="659011caf716165ef8c876b7d5fa2dfe60d04d4c8d32716de9729643c39d18e1" exitCode=0 Nov 03 23:26:18.865150 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:18.864730 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" event={"ID":"36dceb03-a50d-40cc-96d3-f939896bdf06","Type":"ContainerDied","Data":"659011caf716165ef8c876b7d5fa2dfe60d04d4c8d32716de9729643c39d18e1"} Nov 03 23:26:20.004343 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.004312 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:20.146988 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.146896 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-openshift-ca\") pod \"36dceb03-a50d-40cc-96d3-f939896bdf06\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " Nov 03 23:26:20.147190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.147004 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dceb03-a50d-40cc-96d3-f939896bdf06-kube-api-access-xf529\" (UniqueName: \"kubernetes.io/projected/36dceb03-a50d-40cc-96d3-f939896bdf06-kube-api-access-xf529\") pod \"36dceb03-a50d-40cc-96d3-f939896bdf06\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " Nov 03 23:26:20.147190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.147069 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-xks-ca\") pod \"36dceb03-a50d-40cc-96d3-f939896bdf06\" (UID: \"36dceb03-a50d-40cc-96d3-f939896bdf06\") " Nov 03 23:26:20.147339 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.147308 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-openshift-ca" pod "36dceb03-a50d-40cc-96d3-f939896bdf06" (UID: "36dceb03-a50d-40cc-96d3-f939896bdf06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:20.149182 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.149154 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dceb03-a50d-40cc-96d3-f939896bdf06-kube-api-access-xf529" pod "36dceb03-a50d-40cc-96d3-f939896bdf06" (UID: "36dceb03-a50d-40cc-96d3-f939896bdf06"). InnerVolumeSpecName "kube-api-access-xf529". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:20.149182 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.149159 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-xks-ca" pod "36dceb03-a50d-40cc-96d3-f939896bdf06" (UID: "36dceb03-a50d-40cc-96d3-f939896bdf06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:26:20.248597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.248552 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:20.248597 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.248589 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf529\" (UniqueName: \"kubernetes.io/projected/36dceb03-a50d-40cc-96d3-f939896bdf06-kube-api-access-xf529\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:20.248840 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.248605 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dceb03-a50d-40cc-96d3-f939896bdf06-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:20.875868 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.875825 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" event={"ID":"36dceb03-a50d-40cc-96d3-f939896bdf06","Type":"ContainerDied","Data":"98254d52ff9a7d0bfa4c724279edbe7d002916433c3a71ba56a03f55d9ff500e"} Nov 03 23:26:20.876052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.875886 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98254d52ff9a7d0bfa4c724279edbe7d002916433c3a71ba56a03f55d9ff500e" Nov 03 23:26:20.876052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:20.875836 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct" Nov 03 23:26:24.500576 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:26:24.500536 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:26:28.246117 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.246080 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv"] Nov 03 23:26:28.246624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.246503 5272 cpu_manager.go:414] "containerMap: removing container" podUID="36dceb03-a50d-40cc-96d3-f939896bdf06" containerName="revoke-keys" Nov 03 23:26:28.246624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.246513 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dceb03-a50d-40cc-96d3-f939896bdf06" containerName="revoke-keys" Nov 03 23:26:28.246624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.246580 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dceb03-a50d-40cc-96d3-f939896bdf06" containerName="revoke-keys" Nov 03 23:26:28.249699 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.249675 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.325131 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.325088 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgns\" (UniqueName: \"kubernetes.io/projected/52326002-2ea9-43c9-ae88-cab90306c86b-kube-api-access-blgns\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.325317 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.325146 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.325317 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.325242 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.344905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.344875 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:26:28.344905 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.344898 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:26:28.345087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.344898 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:26:28.345087 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.344903 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:26:28.346688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.346669 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv"] Nov 03 23:26:28.426086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.426051 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.426306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.426191 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgns\" (UniqueName: \"kubernetes.io/projected/52326002-2ea9-43c9-ae88-cab90306c86b-kube-api-access-blgns\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.426306 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.426224 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.426900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.426872 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.428406 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.428388 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.483652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.483621 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgns\" (UniqueName: \"kubernetes.io/projected/52326002-2ea9-43c9-ae88-cab90306c86b-kube-api-access-blgns\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.563922 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.563884 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:28.933031 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:28.932949 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv"] Nov 03 23:26:28.940293 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:26:28.940263 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52326002_2ea9_43c9_ae88_cab90306c86b.slice/crio-fb508c9546c1595de50c4d953acbfe6cef1d156198749eefe4bd9b5df6b27283 WatchSource:0}: Error finding container fb508c9546c1595de50c4d953acbfe6cef1d156198749eefe4bd9b5df6b27283: Status 404 returned error can't find the container with id fb508c9546c1595de50c4d953acbfe6cef1d156198749eefe4bd9b5df6b27283 Nov 03 23:26:29.928476 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:29.928437 5272 generic.go:356] "Generic (PLEG): container finished" podID="52326002-2ea9-43c9-ae88-cab90306c86b" containerID="afbb06aeaa4e35f399188124599e23a105e9407e4c08fe1fef5e84e4481d64ff" exitCode=0 Nov 03 23:26:29.928913 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:29.928499 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" event={"ID":"52326002-2ea9-43c9-ae88-cab90306c86b","Type":"ContainerDied","Data":"afbb06aeaa4e35f399188124599e23a105e9407e4c08fe1fef5e84e4481d64ff"} Nov 03 23:26:29.928913 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:29.928549 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" event={"ID":"52326002-2ea9-43c9-ae88-cab90306c86b","Type":"ContainerStarted","Data":"fb508c9546c1595de50c4d953acbfe6cef1d156198749eefe4bd9b5df6b27283"} Nov 03 23:26:31.071931 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.071903 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:31.151070 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.151033 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-xks-ca\") pod \"52326002-2ea9-43c9-ae88-cab90306c86b\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " Nov 03 23:26:31.151280 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.151141 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52326002-2ea9-43c9-ae88-cab90306c86b-kube-api-access-blgns\" (UniqueName: \"kubernetes.io/projected/52326002-2ea9-43c9-ae88-cab90306c86b-kube-api-access-blgns\") pod \"52326002-2ea9-43c9-ae88-cab90306c86b\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " Nov 03 23:26:31.151280 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.151198 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-openshift-ca\") pod \"52326002-2ea9-43c9-ae88-cab90306c86b\" (UID: \"52326002-2ea9-43c9-ae88-cab90306c86b\") " Nov 03 23:26:31.151583 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.151558 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-openshift-ca" pod "52326002-2ea9-43c9-ae88-cab90306c86b" (UID: "52326002-2ea9-43c9-ae88-cab90306c86b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:31.153252 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.153224 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-xks-ca" pod "52326002-2ea9-43c9-ae88-cab90306c86b" (UID: "52326002-2ea9-43c9-ae88-cab90306c86b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:26:31.153357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.153303 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52326002-2ea9-43c9-ae88-cab90306c86b-kube-api-access-blgns" pod "52326002-2ea9-43c9-ae88-cab90306c86b" (UID: "52326002-2ea9-43c9-ae88-cab90306c86b"). InnerVolumeSpecName "kube-api-access-blgns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:31.252320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.252230 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:31.252320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.252260 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgns\" (UniqueName: \"kubernetes.io/projected/52326002-2ea9-43c9-ae88-cab90306c86b-kube-api-access-blgns\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:31.252320 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.252272 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52326002-2ea9-43c9-ae88-cab90306c86b-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:31.871511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.871468 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:26:31.939571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.939531 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" Nov 03 23:26:31.939571 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.939559 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv" event={"ID":"52326002-2ea9-43c9-ae88-cab90306c86b","Type":"ContainerDied","Data":"fb508c9546c1595de50c4d953acbfe6cef1d156198749eefe4bd9b5df6b27283"} Nov 03 23:26:31.939787 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:31.939609 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb508c9546c1595de50c4d953acbfe6cef1d156198749eefe4bd9b5df6b27283" Nov 03 23:26:39.749540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:39.749500 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5"] Nov 03 23:26:39.855248 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:39.855199 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd"] Nov 03 23:26:41.103052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:41.103000 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6"] Nov 03 23:26:41.115362 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:41.115328 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650e8641-2q9n6"] Nov 03 23:26:41.493932 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:41.493238 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4246b50d-11d6-4bee-aacf-45164f0022de" path="/var/lib/kubelet/pods/4246b50d-11d6-4bee-aacf-45164f0022de/volumes" Nov 03 23:26:44.534663 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:44.534614 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv"] Nov 03 23:26:44.581261 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:44.581225 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e433fd-c7vvv"] Nov 03 23:26:44.856243 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:44.856140 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" podUID="60eb8716-e74a-40ca-8330-d8d514a83294" containerName="payload-pre-processing" containerID="cri-o://51b778ba5b19c2fa659ec4491eabcaa8d5fc37135fb0ab1b6270b9af6f041736" gracePeriod=25 Nov 03 23:26:44.980020 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:44.979979 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" podUID="3cbd086a-e80b-4e23-92fc-3aca371b5df2" containerName="payload-processing" containerID="cri-o://48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba" gracePeriod=25 Nov 03 23:26:45.005421 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.005392 5272 generic.go:356] "Generic (PLEG): container finished" podID="60eb8716-e74a-40ca-8330-d8d514a83294" containerID="51b778ba5b19c2fa659ec4491eabcaa8d5fc37135fb0ab1b6270b9af6f041736" exitCode=0 Nov 03 23:26:45.005549 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.005435 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" event={"ID":"60eb8716-e74a-40ca-8330-d8d514a83294","Type":"ContainerDied","Data":"51b778ba5b19c2fa659ec4491eabcaa8d5fc37135fb0ab1b6270b9af6f041736"} Nov 03 23:26:45.106243 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.106218 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:26:45.185254 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.185216 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60eb8716-e74a-40ca-8330-d8d514a83294-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eb8716-e74a-40ca-8330-d8d514a83294-plugins-config-volume\") pod \"60eb8716-e74a-40ca-8330-d8d514a83294\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " Nov 03 23:26:45.185420 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.185268 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60eb8716-e74a-40ca-8330-d8d514a83294-kube-api-access-qnl4z\" (UniqueName: \"kubernetes.io/projected/60eb8716-e74a-40ca-8330-d8d514a83294-kube-api-access-qnl4z\") pod \"60eb8716-e74a-40ca-8330-d8d514a83294\" (UID: \"60eb8716-e74a-40ca-8330-d8d514a83294\") " Nov 03 23:26:45.185665 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.185642 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60eb8716-e74a-40ca-8330-d8d514a83294-plugins-config-volume" pod "60eb8716-e74a-40ca-8330-d8d514a83294" (UID: "60eb8716-e74a-40ca-8330-d8d514a83294"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:45.189011 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.188988 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60eb8716-e74a-40ca-8330-d8d514a83294-kube-api-access-qnl4z" pod "60eb8716-e74a-40ca-8330-d8d514a83294" (UID: "60eb8716-e74a-40ca-8330-d8d514a83294"). InnerVolumeSpecName "kube-api-access-qnl4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:45.218480 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.218454 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:26:45.286737 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.286707 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cbd086a-e80b-4e23-92fc-3aca371b5df2-kube-api-access-gtwzr\" (UniqueName: \"kubernetes.io/projected/3cbd086a-e80b-4e23-92fc-3aca371b5df2-kube-api-access-gtwzr\") pod \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " Nov 03 23:26:45.286923 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.286782 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cbd086a-e80b-4e23-92fc-3aca371b5df2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cbd086a-e80b-4e23-92fc-3aca371b5df2-plugins-config-volume\") pod \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\" (UID: \"3cbd086a-e80b-4e23-92fc-3aca371b5df2\") " Nov 03 23:26:45.287066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.287052 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60eb8716-e74a-40ca-8330-d8d514a83294-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:45.287111 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.287068 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnl4z\" (UniqueName: \"kubernetes.io/projected/60eb8716-e74a-40ca-8330-d8d514a83294-kube-api-access-qnl4z\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:45.287181 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.287147 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbd086a-e80b-4e23-92fc-3aca371b5df2-plugins-config-volume" pod "3cbd086a-e80b-4e23-92fc-3aca371b5df2" (UID: "3cbd086a-e80b-4e23-92fc-3aca371b5df2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:45.288895 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.288871 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbd086a-e80b-4e23-92fc-3aca371b5df2-kube-api-access-gtwzr" pod "3cbd086a-e80b-4e23-92fc-3aca371b5df2" (UID: "3cbd086a-e80b-4e23-92fc-3aca371b5df2"). InnerVolumeSpecName "kube-api-access-gtwzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:45.388527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.388493 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cbd086a-e80b-4e23-92fc-3aca371b5df2-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:45.388527 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.388521 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwzr\" (UniqueName: \"kubernetes.io/projected/3cbd086a-e80b-4e23-92fc-3aca371b5df2-kube-api-access-gtwzr\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:26:45.491565 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:45.491487 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52326002-2ea9-43c9-ae88-cab90306c86b" path="/var/lib/kubelet/pods/52326002-2ea9-43c9-ae88-cab90306c86b/volumes" Nov 03 23:26:46.011214 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.011185 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" Nov 03 23:26:46.011624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.011221 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd" event={"ID":"60eb8716-e74a-40ca-8330-d8d514a83294","Type":"ContainerDied","Data":"dc8b919e7221099cf9a9f9fb8994fde628edf0608583084bdbb7723aeb2441c1"} Nov 03 23:26:46.011624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.011280 5272 scope.go:122] "RemoveContainer" containerID="51b778ba5b19c2fa659ec4491eabcaa8d5fc37135fb0ab1b6270b9af6f041736" Nov 03 23:26:46.012857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.012639 5272 generic.go:356] "Generic (PLEG): container finished" podID="3cbd086a-e80b-4e23-92fc-3aca371b5df2" containerID="48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba" exitCode=0 Nov 03 23:26:46.012857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.012678 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" Nov 03 23:26:46.012857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.012677 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" event={"ID":"3cbd086a-e80b-4e23-92fc-3aca371b5df2","Type":"ContainerDied","Data":"48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba"} Nov 03 23:26:46.012857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.012813 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5" event={"ID":"3cbd086a-e80b-4e23-92fc-3aca371b5df2","Type":"ContainerDied","Data":"c1f32a230b1071cf007c4b45c52d2c6cddd78f6967eb8212313641fab255d1bb"} Nov 03 23:26:46.020156 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.020134 5272 scope.go:122] "RemoveContainer" containerID="48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba" Nov 03 23:26:46.029575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.029540 5272 scope.go:122] "RemoveContainer" containerID="48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba" Nov 03 23:26:46.029922 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:26:46.029901 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba\": container with ID starting with 48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba not found: ID does not exist" containerID="48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba" Nov 03 23:26:46.030000 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.029929 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba"} err="failed to get container status \"48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba\": rpc error: code = NotFound desc = could not find container \"48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba\": container with ID starting with 48510aa2ee31f692a541cedf9644c31cdf12650e8950c54842a4b43cefc84aba not found: ID does not exist" Nov 03 23:26:46.163449 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.163367 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd"] Nov 03 23:26:46.210172 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.210138 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e433fd-5df9f8pxkd"] Nov 03 23:26:46.269877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.269844 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5"] Nov 03 23:26:46.276866 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:46.276832 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e433fd-7c6897ff52rsr5"] Nov 03 23:26:47.173002 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:47.172549 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:26:47.490849 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:47.490738 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cbd086a-e80b-4e23-92fc-3aca371b5df2" path="/var/lib/kubelet/pods/3cbd086a-e80b-4e23-92fc-3aca371b5df2/volumes" Nov 03 23:26:47.491236 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:26:47.491215 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60eb8716-e74a-40ca-8330-d8d514a83294" path="/var/lib/kubelet/pods/60eb8716-e74a-40ca-8330-d8d514a83294/volumes" Nov 03 23:26:54.500176 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:26:54.500089 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:27:01.768046 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:27:01.768008 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:27:20.045303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:27:20.045263 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d"] Nov 03 23:27:20.047885 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:27:20.047857 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksv8d"] Nov 03 23:27:21.491642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:27:21.491605 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52e3e69-087b-4ce0-b384-e58b0b0e399c" path="/var/lib/kubelet/pods/e52e3e69-087b-4ce0-b384-e58b0b0e399c/volumes" Nov 03 23:27:24.499918 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:27:24.499881 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:27:44.614462 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:27:44.614427 5272 scope.go:122] "RemoveContainer" containerID="980520e3177a9d2a7e9f953daf981267fc297bbf6000bc66468f44174a4fc157" Nov 03 23:27:47.067746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:27:47.066658 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:27:54.500387 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:27:54.500269 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:28:17.790307 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:17.790272 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:28:24.499685 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:28:24.499601 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:28:24.499685 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.499640 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp"] Nov 03 23:28:24.500096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500008 5272 cpu_manager.go:414] "containerMap: removing container" podUID="60eb8716-e74a-40ca-8330-d8d514a83294" containerName="payload-pre-processing" Nov 03 23:28:24.500096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500020 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60eb8716-e74a-40ca-8330-d8d514a83294" containerName="payload-pre-processing" Nov 03 23:28:24.500096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500045 5272 cpu_manager.go:414] "containerMap: removing container" podUID="52326002-2ea9-43c9-ae88-cab90306c86b" containerName="revoke-keys" Nov 03 23:28:24.500096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500050 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52326002-2ea9-43c9-ae88-cab90306c86b" containerName="revoke-keys" Nov 03 23:28:24.500096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500059 5272 cpu_manager.go:414] "containerMap: removing container" podUID="3cbd086a-e80b-4e23-92fc-3aca371b5df2" containerName="payload-processing" Nov 03 23:28:24.500096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500064 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cbd086a-e80b-4e23-92fc-3aca371b5df2" containerName="payload-processing" Nov 03 23:28:24.500289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500124 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="60eb8716-e74a-40ca-8330-d8d514a83294" containerName="payload-pre-processing" Nov 03 23:28:24.500289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500136 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="52326002-2ea9-43c9-ae88-cab90306c86b" containerName="revoke-keys" Nov 03 23:28:24.500289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.500142 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cbd086a-e80b-4e23-92fc-3aca371b5df2" containerName="payload-processing" Nov 03 23:28:24.503332 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.503310 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.598704 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.598656 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:28:24.598704 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.598656 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:28:24.598937 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.598659 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:28:24.598937 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.598672 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:28:24.600287 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.600267 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp"] Nov 03 23:28:24.612199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.612179 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.612283 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.612236 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9tw\" (UniqueName: \"kubernetes.io/projected/ac8b7820-db4d-49fc-997d-299d93ea623e-kube-api-access-9x9tw\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.612345 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.612327 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.713502 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.713465 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.713692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.713611 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9tw\" (UniqueName: \"kubernetes.io/projected/ac8b7820-db4d-49fc-997d-299d93ea623e-kube-api-access-9x9tw\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.713692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.713655 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.714180 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.714157 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.715919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.715897 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.771208 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.771148 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9tw\" (UniqueName: \"kubernetes.io/projected/ac8b7820-db4d-49fc-997d-299d93ea623e-kube-api-access-9x9tw\") pod \"maas-api-revoke-sub-452b6d7b-sqfnp\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:24.815620 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:24.815590 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:25.183927 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:25.183893 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp"] Nov 03 23:28:25.191234 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:28:25.191204 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8b7820_db4d_49fc_997d_299d93ea623e.slice/crio-83ece39aa26cf29932856899bc9ad6cf3aa2b6f1d8079bafec2bba8fe6ef0bcf WatchSource:0}: Error finding container 83ece39aa26cf29932856899bc9ad6cf3aa2b6f1d8079bafec2bba8fe6ef0bcf: Status 404 returned error can't find the container with id 83ece39aa26cf29932856899bc9ad6cf3aa2b6f1d8079bafec2bba8fe6ef0bcf Nov 03 23:28:25.578050 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:25.578008 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" event={"ID":"ac8b7820-db4d-49fc-997d-299d93ea623e","Type":"ContainerStarted","Data":"81885b001c77c704d01df1a5a5b441f95472c790f63b6c89db0481a6945218ce"} Nov 03 23:28:25.578461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:25.578071 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" event={"ID":"ac8b7820-db4d-49fc-997d-299d93ea623e","Type":"ContainerStarted","Data":"83ece39aa26cf29932856899bc9ad6cf3aa2b6f1d8079bafec2bba8fe6ef0bcf"} Nov 03 23:28:25.686887 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:25.686799 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" podStartSLOduration=1.686782027 podStartE2EDuration="1.686782027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:25.686231087 +0000 UTC m=+3886.308539398" watchObservedRunningTime="2026-11-03 23:28:25.686782027 +0000 UTC m=+3886.309090334" Nov 03 23:28:26.584500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:26.584459 5272 generic.go:356] "Generic (PLEG): container finished" podID="ac8b7820-db4d-49fc-997d-299d93ea623e" containerID="81885b001c77c704d01df1a5a5b441f95472c790f63b6c89db0481a6945218ce" exitCode=0 Nov 03 23:28:26.584950 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:26.584538 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" event={"ID":"ac8b7820-db4d-49fc-997d-299d93ea623e","Type":"ContainerDied","Data":"81885b001c77c704d01df1a5a5b441f95472c790f63b6c89db0481a6945218ce"} Nov 03 23:28:27.725488 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.725461 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:27.844837 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.844714 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-openshift-ca\") pod \"ac8b7820-db4d-49fc-997d-299d93ea623e\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " Nov 03 23:28:27.844837 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.844829 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-xks-ca\") pod \"ac8b7820-db4d-49fc-997d-299d93ea623e\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " Nov 03 23:28:27.845094 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.844923 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8b7820-db4d-49fc-997d-299d93ea623e-kube-api-access-9x9tw\" (UniqueName: \"kubernetes.io/projected/ac8b7820-db4d-49fc-997d-299d93ea623e-kube-api-access-9x9tw\") pod \"ac8b7820-db4d-49fc-997d-299d93ea623e\" (UID: \"ac8b7820-db4d-49fc-997d-299d93ea623e\") " Nov 03 23:28:27.845207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.845176 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-openshift-ca" pod "ac8b7820-db4d-49fc-997d-299d93ea623e" (UID: "ac8b7820-db4d-49fc-997d-299d93ea623e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:28:27.847044 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.847001 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-xks-ca" pod "ac8b7820-db4d-49fc-997d-299d93ea623e" (UID: "ac8b7820-db4d-49fc-997d-299d93ea623e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:28:27.847192 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.847079 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8b7820-db4d-49fc-997d-299d93ea623e-kube-api-access-9x9tw" pod "ac8b7820-db4d-49fc-997d-299d93ea623e" (UID: "ac8b7820-db4d-49fc-997d-299d93ea623e"). InnerVolumeSpecName "kube-api-access-9x9tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:28:27.945992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.945957 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:28:27.945992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.945989 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x9tw\" (UniqueName: \"kubernetes.io/projected/ac8b7820-db4d-49fc-997d-299d93ea623e-kube-api-access-9x9tw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:28:27.945992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:27.946000 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b7820-db4d-49fc-997d-299d93ea623e-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:28:28.594963 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:28.594932 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" Nov 03 23:28:28.594963 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:28.594946 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp" event={"ID":"ac8b7820-db4d-49fc-997d-299d93ea623e","Type":"ContainerDied","Data":"83ece39aa26cf29932856899bc9ad6cf3aa2b6f1d8079bafec2bba8fe6ef0bcf"} Nov 03 23:28:28.595187 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:28.594996 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ece39aa26cf29932856899bc9ad6cf3aa2b6f1d8079bafec2bba8fe6ef0bcf" Nov 03 23:28:37.753543 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:37.753508 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:28:39.934298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:39.934257 5272 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal" podUID="d79e0f53b5b82fd1a135f4636735856a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:28:44.647122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:44.647083 5272 scope.go:122] "RemoveContainer" containerID="37fc7c0c7c39d73931dd9051929cbad1cb77dffa60e9f81b75aa92d4aeff5cdf" Nov 03 23:28:44.708085 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:44.704663 5272 scope.go:122] "RemoveContainer" containerID="6e35284936cee85422709489a6d3095993995f4198a59a4c4d27de2b785bf52f" Nov 03 23:28:48.748758 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:48.748725 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:28:54.498762 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:28:54.498720 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:28:59.620294 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.620250 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s"] Nov 03 23:28:59.620788 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.620602 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ac8b7820-db4d-49fc-997d-299d93ea623e" containerName="revoke-keys" Nov 03 23:28:59.620788 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.620614 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8b7820-db4d-49fc-997d-299d93ea623e" containerName="revoke-keys" Nov 03 23:28:59.620788 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.620685 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8b7820-db4d-49fc-997d-299d93ea623e" containerName="revoke-keys" Nov 03 23:28:59.625059 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.625033 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.683021 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.682983 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.683203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.683057 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.683203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.683110 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgpf\" (UniqueName: \"kubernetes.io/projected/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-kube-api-access-6jgpf\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.722421 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.722385 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:28:59.722586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.722386 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:28:59.722586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.722453 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:28:59.722586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.722534 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:28:59.724403 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.724376 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s"] Nov 03 23:28:59.784253 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.784221 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.784434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.784283 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.784434 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.784328 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgpf\" (UniqueName: \"kubernetes.io/projected/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-kube-api-access-6jgpf\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.784982 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.784957 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.786655 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.786636 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.842130 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.842101 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgpf\" (UniqueName: \"kubernetes.io/projected/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-kube-api-access-6jgpf\") pod \"maas-api-revoke-sub-3e31a506-8c45s\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:28:59.939692 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:28:59.939633 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:29:00.311869 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:00.311842 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s"] Nov 03 23:29:00.319504 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:29:00.319478 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d58f4c_4bc1_45d4_a458_ba2cd3f2feff.slice/crio-73c95594b0761380ddcb535d4372bb5fee991918387b886fae1c98f0705b7218 WatchSource:0}: Error finding container 73c95594b0761380ddcb535d4372bb5fee991918387b886fae1c98f0705b7218: Status 404 returned error can't find the container with id 73c95594b0761380ddcb535d4372bb5fee991918387b886fae1c98f0705b7218 Nov 03 23:29:00.775891 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:00.775852 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" event={"ID":"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff","Type":"ContainerStarted","Data":"fe81cea999602c60470851a29ef84e5c562f6aee5d4610c027369feccce1f615"} Nov 03 23:29:00.776210 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:00.775917 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" event={"ID":"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff","Type":"ContainerStarted","Data":"73c95594b0761380ddcb535d4372bb5fee991918387b886fae1c98f0705b7218"} Nov 03 23:29:00.884511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:00.884463 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" podStartSLOduration=1.884449936 podStartE2EDuration="1.884449936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:00.883668874 +0000 UTC m=+3921.505977180" watchObservedRunningTime="2026-11-03 23:29:00.884449936 +0000 UTC m=+3921.506758242" Nov 03 23:29:01.782311 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:01.782274 5272 generic.go:356] "Generic (PLEG): container finished" podID="23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" containerID="fe81cea999602c60470851a29ef84e5c562f6aee5d4610c027369feccce1f615" exitCode=0 Nov 03 23:29:01.782695 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:01.782324 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" event={"ID":"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff","Type":"ContainerDied","Data":"fe81cea999602c60470851a29ef84e5c562f6aee5d4610c027369feccce1f615"} Nov 03 23:29:02.443815 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:02.443772 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:29:02.952058 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:02.951752 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:29:03.014655 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.014606 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-kube-api-access-6jgpf\" (UniqueName: \"kubernetes.io/projected/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-kube-api-access-6jgpf\") pod \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " Nov 03 23:29:03.014881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.014683 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-xks-ca\") pod \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " Nov 03 23:29:03.014881 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.014731 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-openshift-ca\") pod \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\" (UID: \"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff\") " Nov 03 23:29:03.015364 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.015325 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-openshift-ca" pod "23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" (UID: "23d58f4c-4bc1-45d4-a458-ba2cd3f2feff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:29:03.017424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.017392 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-xks-ca" pod "23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" (UID: "23d58f4c-4bc1-45d4-a458-ba2cd3f2feff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:29:03.017687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.017663 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-kube-api-access-6jgpf" pod "23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" (UID: "23d58f4c-4bc1-45d4-a458-ba2cd3f2feff"). InnerVolumeSpecName "kube-api-access-6jgpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:29:03.115709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.115609 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jgpf\" (UniqueName: \"kubernetes.io/projected/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-kube-api-access-6jgpf\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:03.115709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.115650 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:03.115709 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.115665 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:03.813914 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.811859 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" event={"ID":"23d58f4c-4bc1-45d4-a458-ba2cd3f2feff","Type":"ContainerDied","Data":"73c95594b0761380ddcb535d4372bb5fee991918387b886fae1c98f0705b7218"} Nov 03 23:29:03.813914 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.811920 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s" Nov 03 23:29:03.813914 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:03.811940 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c95594b0761380ddcb535d4372bb5fee991918387b886fae1c98f0705b7218" Nov 03 23:29:13.888235 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:13.888201 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:29:14.057308 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:14.057280 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z"] Nov 03 23:29:14.103873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:14.103836 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vzf7z"] Nov 03 23:29:15.498661 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:15.498624 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d83b70-c34e-449e-9795-7324e03dff06" path="/var/lib/kubelet/pods/61d83b70-c34e-449e-9795-7324e03dff06/volumes" Nov 03 23:29:23.993732 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:23.993701 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:29:24.499095 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:29:24.499058 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:29:32.179271 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.179237 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh"] Nov 03 23:29:32.179645 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.179626 5272 cpu_manager.go:414] "containerMap: removing container" podUID="23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" containerName="revoke-keys" Nov 03 23:29:32.179645 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.179637 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" containerName="revoke-keys" Nov 03 23:29:32.179723 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.179699 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" containerName="revoke-keys" Nov 03 23:29:32.183066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.183044 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.279337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.279308 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:29:32.279337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.279318 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:29:32.279539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.279341 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:29:32.279539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.279399 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:29:32.280898 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.280878 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh"] Nov 03 23:29:32.301746 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.301719 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.301898 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.301765 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6h6g\" (UniqueName: \"kubernetes.io/projected/521f94a0-4fac-436e-9ca4-20faa32666bc-kube-api-access-x6h6g\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.301898 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.301880 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.403119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.403094 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.403242 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.403138 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6h6g\" (UniqueName: \"kubernetes.io/projected/521f94a0-4fac-436e-9ca4-20faa32666bc-kube-api-access-x6h6g\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.403242 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.403177 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.403727 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.403710 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.405465 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.405445 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.460026 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.459972 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6h6g\" (UniqueName: \"kubernetes.io/projected/521f94a0-4fac-436e-9ca4-20faa32666bc-kube-api-access-x6h6g\") pod \"maas-api-revoke-sub-da690c1c-khvkh\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.495190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.495168 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:32.861958 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.861921 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh"] Nov 03 23:29:32.869173 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:29:32.869145 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521f94a0_4fac_436e_9ca4_20faa32666bc.slice/crio-d73c55e24e045d0d8296b32668e0c6996a8b98b1a37ede891ce26298a619aec7 WatchSource:0}: Error finding container d73c55e24e045d0d8296b32668e0c6996a8b98b1a37ede891ce26298a619aec7: Status 404 returned error can't find the container with id d73c55e24e045d0d8296b32668e0c6996a8b98b1a37ede891ce26298a619aec7 Nov 03 23:29:32.997475 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:32.997445 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" event={"ID":"521f94a0-4fac-436e-9ca4-20faa32666bc","Type":"ContainerStarted","Data":"d73c55e24e045d0d8296b32668e0c6996a8b98b1a37ede891ce26298a619aec7"} Nov 03 23:29:34.003498 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:34.003427 5272 generic.go:356] "Generic (PLEG): container finished" podID="521f94a0-4fac-436e-9ca4-20faa32666bc" containerID="3c36e52cff759cd0c4e6648eb8c1477cb799842c8f997cf77a5d57029e63e49a" exitCode=0 Nov 03 23:29:34.003498 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:34.003495 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" event={"ID":"521f94a0-4fac-436e-9ca4-20faa32666bc","Type":"ContainerDied","Data":"3c36e52cff759cd0c4e6648eb8c1477cb799842c8f997cf77a5d57029e63e49a"} Nov 03 23:29:35.143140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.143116 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:35.330332 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.330301 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-xks-ca\") pod \"521f94a0-4fac-436e-9ca4-20faa32666bc\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " Nov 03 23:29:35.330534 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.330424 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-openshift-ca\") pod \"521f94a0-4fac-436e-9ca4-20faa32666bc\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " Nov 03 23:29:35.330534 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.330451 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521f94a0-4fac-436e-9ca4-20faa32666bc-kube-api-access-x6h6g\" (UniqueName: \"kubernetes.io/projected/521f94a0-4fac-436e-9ca4-20faa32666bc-kube-api-access-x6h6g\") pod \"521f94a0-4fac-436e-9ca4-20faa32666bc\" (UID: \"521f94a0-4fac-436e-9ca4-20faa32666bc\") " Nov 03 23:29:35.330857 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.330796 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-openshift-ca" pod "521f94a0-4fac-436e-9ca4-20faa32666bc" (UID: "521f94a0-4fac-436e-9ca4-20faa32666bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:29:35.332378 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.332347 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-xks-ca" pod "521f94a0-4fac-436e-9ca4-20faa32666bc" (UID: "521f94a0-4fac-436e-9ca4-20faa32666bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:29:35.332544 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.332522 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521f94a0-4fac-436e-9ca4-20faa32666bc-kube-api-access-x6h6g" pod "521f94a0-4fac-436e-9ca4-20faa32666bc" (UID: "521f94a0-4fac-436e-9ca4-20faa32666bc"). InnerVolumeSpecName "kube-api-access-x6h6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:29:35.431339 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.431307 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:35.431339 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.431335 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521f94a0-4fac-436e-9ca4-20faa32666bc-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:35.431339 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:35.431347 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6h6g\" (UniqueName: \"kubernetes.io/projected/521f94a0-4fac-436e-9ca4-20faa32666bc-kube-api-access-x6h6g\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:29:36.014837 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:36.014791 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" event={"ID":"521f94a0-4fac-436e-9ca4-20faa32666bc","Type":"ContainerDied","Data":"d73c55e24e045d0d8296b32668e0c6996a8b98b1a37ede891ce26298a619aec7"} Nov 03 23:29:36.014837 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:36.014821 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh" Nov 03 23:29:36.015057 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:36.014854 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d73c55e24e045d0d8296b32668e0c6996a8b98b1a37ede891ce26298a619aec7" Nov 03 23:29:46.044645 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:46.044609 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947"] Nov 03 23:29:46.090484 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:46.090456 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ct947"] Nov 03 23:29:47.490752 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:47.490718 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37c5062-c189-4908-98cf-a5ab4c1f0b60" path="/var/lib/kubelet/pods/b37c5062-c189-4908-98cf-a5ab4c1f0b60/volumes" Nov 03 23:29:48.740892 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:48.740858 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:29:52.344794 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.344712 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz"] Nov 03 23:29:52.345179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.345113 5272 cpu_manager.go:414] "containerMap: removing container" podUID="521f94a0-4fac-436e-9ca4-20faa32666bc" containerName="revoke-keys" Nov 03 23:29:52.345179 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.345126 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521f94a0-4fac-436e-9ca4-20faa32666bc" containerName="revoke-keys" Nov 03 23:29:52.345256 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.345211 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="521f94a0-4fac-436e-9ca4-20faa32666bc" containerName="revoke-keys" Nov 03 23:29:52.349135 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.349111 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372431 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372408 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istiod-ca-cert\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372566 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372440 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-socket\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372574 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-data\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372597 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-envoy\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372618 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372615 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-certs\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372635 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqx85\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-kube-api-access-zqx85\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372693 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-token\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.372772 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372749 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-credential-socket\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.373014 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.372787 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-podinfo\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.398193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.398166 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cc1c5661-openshift-default-dockercfg-gp9q6\"" Nov 03 23:29:52.446467 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.446437 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz"] Nov 03 23:29:52.474324 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474292 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-credential-socket\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474330 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-podinfo\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474464 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474362 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istiod-ca-cert\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474584 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474457 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-socket\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474590 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-data\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474649 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474627 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-envoy\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474757 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474660 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-certs\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474757 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474694 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqx85\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-kube-api-access-zqx85\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474757 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474725 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-token\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474757 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-credential-socket\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.474943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.474846 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-socket\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.475193 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.475162 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istiod-ca-cert\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.475307 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.475286 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-data\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.475382 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.475301 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-certs\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.476854 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.476826 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-envoy\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.476954 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.476861 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-podinfo\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.535836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.535748 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-token\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.541485 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.541449 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqx85\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-kube-api-access-zqx85\") pod \"e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:52.661539 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:52.661456 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:53.085223 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:53.085182 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz"] Nov 03 23:29:53.092512 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:29:53.092484 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665cc4fc_02b9_4455_880b_c4cb014c6f3d.slice/crio-defea411350805303490e448d5665d0a88ff84bf83f32adf6a51277f0ac59087 WatchSource:0}: Error finding container defea411350805303490e448d5665d0a88ff84bf83f32adf6a51277f0ac59087: Status 404 returned error can't find the container with id defea411350805303490e448d5665d0a88ff84bf83f32adf6a51277f0ac59087 Nov 03 23:29:53.094492 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:53.094459 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:29:53.094565 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:53.094528 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:29:53.094606 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:53.094579 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:29:54.096729 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:54.096697 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" event={"ID":"665cc4fc-02b9-4455-880b-c4cb014c6f3d","Type":"ContainerStarted","Data":"f5945af27ee107c18b506102c0a9dd94c8bf5055163954964e996c370a84802b"} Nov 03 23:29:54.097168 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:54.096753 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" event={"ID":"665cc4fc-02b9-4455-880b-c4cb014c6f3d","Type":"ContainerStarted","Data":"defea411350805303490e448d5665d0a88ff84bf83f32adf6a51277f0ac59087"} Nov 03 23:29:54.256953 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:54.256867 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podStartSLOduration=2.256850137 podStartE2EDuration="2.256850137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:54.25563538 +0000 UTC m=+3974.877943688" watchObservedRunningTime="2026-11-03 23:29:54.256850137 +0000 UTC m=+3974.879158446" Nov 03 23:29:54.498497 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:29:54.498414 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:29:54.662601 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:54.662562 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:54.667216 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:54.667195 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:55.101625 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:55.101599 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:29:55.102453 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:29:55.102439 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:30:00.165063 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.165026 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw"] Nov 03 23:30:00.168889 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.168863 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.227551 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.227519 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw"] Nov 03 23:30:00.243165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.243137 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895810-jtffw\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.243289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.243186 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhw7\" (UniqueName: \"kubernetes.io/projected/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-kube-api-access-llhw7\") pod \"maas-api-key-cleanup-29895810-jtffw\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.289242 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.289215 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:30:00.320717 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.320689 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:30:00.320877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.320727 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cjll9\"" Nov 03 23:30:00.322585 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.322567 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b"] Nov 03 23:30:00.325949 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.325935 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.344548 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.344526 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895810-jtffw\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.344658 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.344571 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhw7\" (UniqueName: \"kubernetes.io/projected/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-kube-api-access-llhw7\") pod \"maas-api-key-cleanup-29895810-jtffw\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.345119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.345099 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895810-jtffw\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.363619 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.363595 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b"] Nov 03 23:30:00.389493 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.389471 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cc1c5661\"" Nov 03 23:30:00.389654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.389640 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cc1c5661-dockercfg-kjzwp\"" Nov 03 23:30:00.401442 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.401419 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhw7\" (UniqueName: \"kubernetes.io/projected/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-kube-api-access-llhw7\") pod \"maas-api-key-cleanup-29895810-jtffw\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.446109 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.446041 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khs8z\" (UniqueName: \"kubernetes.io/projected/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-kube-api-access-khs8z\") pod \"payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.446229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.446152 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.481943 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.481913 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:00.574335 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.574299 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.575166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.575134 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.575316 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.575296 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khs8z\" (UniqueName: \"kubernetes.io/projected/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-kube-api-access-khs8z\") pod \"payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.632945 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.632916 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khs8z\" (UniqueName: \"kubernetes.io/projected/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-kube-api-access-khs8z\") pod \"payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.638677 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.638648 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:00.849194 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:00.849163 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw"] Nov 03 23:30:00.856528 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:30:00.856500 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8b1ca0_7e2e_42bd_bbd1_cb42e6043372.slice/crio-eee2081d71794e06a23f10852e344b8e081a8deb98f57d7e9dc2179a47f21429 WatchSource:0}: Error finding container eee2081d71794e06a23f10852e344b8e081a8deb98f57d7e9dc2179a47f21429: Status 404 returned error can't find the container with id eee2081d71794e06a23f10852e344b8e081a8deb98f57d7e9dc2179a47f21429 Nov 03 23:30:01.010523 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:01.010493 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b"] Nov 03 23:30:01.017107 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:30:01.017083 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2ec4fd_5ba9_4e5d_b319_92c18699df53.slice/crio-d0a6767c76f4a3ae1c70ce177a5671334f0f24e7373a528d2d97862a43042532 WatchSource:0}: Error finding container d0a6767c76f4a3ae1c70ce177a5671334f0f24e7373a528d2d97862a43042532: Status 404 returned error can't find the container with id d0a6767c76f4a3ae1c70ce177a5671334f0f24e7373a528d2d97862a43042532 Nov 03 23:30:01.145525 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:01.145493 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" event={"ID":"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372","Type":"ContainerStarted","Data":"eee2081d71794e06a23f10852e344b8e081a8deb98f57d7e9dc2179a47f21429"} Nov 03 23:30:01.146948 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:01.146925 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" event={"ID":"ad2ec4fd-5ba9-4e5d-b319-92c18699df53","Type":"ContainerStarted","Data":"d0a6767c76f4a3ae1c70ce177a5671334f0f24e7373a528d2d97862a43042532"} Nov 03 23:30:02.152696 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:02.152662 5272 generic.go:356] "Generic (PLEG): container finished" podID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerID="f38a191c1492c2bd6bdc3e686d60c686c3f27da1c5005b920cc5b8c1d12dc6ca" exitCode=60 Nov 03 23:30:02.153178 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:02.152750 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" event={"ID":"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372","Type":"ContainerDied","Data":"f38a191c1492c2bd6bdc3e686d60c686c3f27da1c5005b920cc5b8c1d12dc6ca"} Nov 03 23:30:02.153178 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:02.152956 5272 scope.go:122] "RemoveContainer" containerID="f38a191c1492c2bd6bdc3e686d60c686c3f27da1c5005b920cc5b8c1d12dc6ca" Nov 03 23:30:02.154370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:02.154341 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" event={"ID":"ad2ec4fd-5ba9-4e5d-b319-92c18699df53","Type":"ContainerStarted","Data":"b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c"} Nov 03 23:30:02.154444 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:02.154381 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:02.369590 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:02.369545 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" podStartSLOduration=2.369532776 podStartE2EDuration="2.369532776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:30:02.368443807 +0000 UTC m=+3982.990752114" watchObservedRunningTime="2026-11-03 23:30:02.369532776 +0000 UTC m=+3982.991841076" Nov 03 23:30:03.162052 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:03.162020 5272 generic.go:356] "Generic (PLEG): container finished" podID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerID="92b4f29e1472b0bf6b1a6ce442097273d5afb2a7bc844ca741ca2e1651bf9654" exitCode=60 Nov 03 23:30:03.162473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:03.162104 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" event={"ID":"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372","Type":"ContainerDied","Data":"92b4f29e1472b0bf6b1a6ce442097273d5afb2a7bc844ca741ca2e1651bf9654"} Nov 03 23:30:03.162473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:03.162186 5272 scope.go:122] "RemoveContainer" containerID="f38a191c1492c2bd6bdc3e686d60c686c3f27da1c5005b920cc5b8c1d12dc6ca" Nov 03 23:30:03.162473 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:03.162371 5272 scope.go:122] "RemoveContainer" containerID="92b4f29e1472b0bf6b1a6ce442097273d5afb2a7bc844ca741ca2e1651bf9654" Nov 03 23:30:03.162630 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:30:03.162611 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895810-jtffw_odh-ai-gateway-infra(ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" Nov 03 23:30:04.168738 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:04.168709 5272 scope.go:122] "RemoveContainer" containerID="92b4f29e1472b0bf6b1a6ce442097273d5afb2a7bc844ca741ca2e1651bf9654" Nov 03 23:30:04.169199 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:30:04.168982 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895810-jtffw_odh-ai-gateway-infra(ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" Nov 03 23:30:09.578428 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:09.578388 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:30:13.168664 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:13.168631 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:17.486376 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:17.486335 5272 scope.go:122] "RemoveContainer" containerID="92b4f29e1472b0bf6b1a6ce442097273d5afb2a7bc844ca741ca2e1651bf9654" Nov 03 23:30:18.243058 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:18.243025 5272 generic.go:356] "Generic (PLEG): container finished" podID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerID="149269a2ac243bb4e867023ed474704f7b9ff4b042fe158f706a08a08bb34583" exitCode=60 Nov 03 23:30:18.243231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:18.243096 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" event={"ID":"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372","Type":"ContainerDied","Data":"149269a2ac243bb4e867023ed474704f7b9ff4b042fe158f706a08a08bb34583"} Nov 03 23:30:18.243231 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:18.243168 5272 scope.go:122] "RemoveContainer" containerID="92b4f29e1472b0bf6b1a6ce442097273d5afb2a7bc844ca741ca2e1651bf9654" Nov 03 23:30:18.243570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:18.243518 5272 scope.go:122] "RemoveContainer" containerID="149269a2ac243bb4e867023ed474704f7b9ff4b042fe158f706a08a08bb34583" Nov 03 23:30:18.243832 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:30:18.243793 5272 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895810-jtffw_odh-ai-gateway-infra(ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" Nov 03 23:30:19.250035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.249924 5272 generic.go:356] "Generic (PLEG): container finished" podID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerID="f5945af27ee107c18b506102c0a9dd94c8bf5055163954964e996c370a84802b" exitCode=0 Nov 03 23:30:19.250035 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.250000 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" event={"ID":"665cc4fc-02b9-4455-880b-c4cb014c6f3d","Type":"ContainerDied","Data":"f5945af27ee107c18b506102c0a9dd94c8bf5055163954964e996c370a84802b"} Nov 03 23:30:19.250570 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.250407 5272 scope.go:122] "RemoveContainer" containerID="f5945af27ee107c18b506102c0a9dd94c8bf5055163954964e996c370a84802b" Nov 03 23:30:19.251776 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.251745 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:30:19.251927 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.251839 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:30:19.251927 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.251874 5272 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:30:19.360953 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.360921 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw"] Nov 03 23:30:19.501506 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.501443 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:19.570070 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.570033 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-maas-api-service-ca\") pod \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " Nov 03 23:30:19.570244 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.570089 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-kube-api-access-llhw7\" (UniqueName: \"kubernetes.io/projected/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-kube-api-access-llhw7\") pod \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\" (UID: \"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372\") " Nov 03 23:30:19.570416 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.570388 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-maas-api-service-ca" pod "ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" (UID: "ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:19.570556 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.570502 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-maas-api-service-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:19.572317 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.572296 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-kube-api-access-llhw7" pod "ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" (UID: "ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372"). InnerVolumeSpecName "kube-api-access-llhw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:19.676918 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:19.672598 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhw7\" (UniqueName: \"kubernetes.io/projected/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372-kube-api-access-llhw7\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:20.273259 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.273221 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" event={"ID":"665cc4fc-02b9-4455-880b-c4cb014c6f3d","Type":"ContainerStarted","Data":"8fd8fbe0d5278d95674ae5ae749c0bbf200f1d98fbac45372ce6081cf43771ec"} Nov 03 23:30:20.274942 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.274913 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" Nov 03 23:30:20.275073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.274938 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw" event={"ID":"ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372","Type":"ContainerDied","Data":"eee2081d71794e06a23f10852e344b8e081a8deb98f57d7e9dc2179a47f21429"} Nov 03 23:30:20.275073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.274991 5272 scope.go:122] "RemoveContainer" containerID="149269a2ac243bb4e867023ed474704f7b9ff4b042fe158f706a08a08bb34583" Nov 03 23:30:20.494183 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.494148 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw"] Nov 03 23:30:20.498429 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.498401 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-jtffw"] Nov 03 23:30:20.662825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.662171 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:30:20.668769 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.668739 5272 patch_prober.go:29] interesting pod/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 03 23:30:20.668951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:20.668795 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 03 23:30:21.491520 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:21.491487 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" path="/var/lib/kubelet/pods/ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372/volumes" Nov 03 23:30:21.665827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:21.665109 5272 patch_prober.go:29] interesting pod/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 03 23:30:21.665827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:21.665169 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 03 23:30:22.668194 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:22.668155 5272 patch_prober.go:29] interesting pod/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 03 23:30:22.668714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:22.668209 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 03 23:30:23.668196 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:23.668152 5272 patch_prober.go:29] interesting pod/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 03 23:30:23.668357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:23.668216 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 03 23:30:24.499069 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:30:24.499034 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:30:24.666827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:24.666118 5272 patch_prober.go:29] interesting pod/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 03 23:30:24.666827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:24.666175 5272 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 03 23:30:25.102370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:25.102330 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:30:25.666219 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:25.666188 5272 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:30:25.667350 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:25.667331 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:30:32.539333 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.539297 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8"] Nov 03 23:30:32.539896 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.539875 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:30:32.540004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.539899 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:30:32.540004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.539916 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:30:32.540004 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.539923 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:30:32.540170 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.540058 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:30:32.540170 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.540078 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:30:32.543624 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.543603 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.589398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.589374 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.589511 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.589442 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4l4g\" (UniqueName: \"kubernetes.io/projected/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-kube-api-access-j4l4g\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.589760 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.589740 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.602828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.600993 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8"] Nov 03 23:30:32.642126 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.642090 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:30:32.642313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.642155 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:30:32.642313 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.642193 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fmpj4\"" Nov 03 23:30:32.642415 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.642299 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:30:32.690416 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.690392 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.690542 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.690436 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4l4g\" (UniqueName: \"kubernetes.io/projected/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-kube-api-access-j4l4g\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.690542 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.690489 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.691101 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.691075 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.692834 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.692816 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.748048 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.748026 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4l4g\" (UniqueName: \"kubernetes.io/projected/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-kube-api-access-j4l4g\") pod \"maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:32.854966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:32.854912 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:33.225131 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:33.225048 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8"] Nov 03 23:30:33.231744 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:30:33.231716 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a5c581_c755_4d1a_8f7f_7f1bcd227c8d.slice/crio-9347604974e79d5ca3d674dcceb12464cf8ea43742a7aa08a64d2bb7edcfbc85 WatchSource:0}: Error finding container 9347604974e79d5ca3d674dcceb12464cf8ea43742a7aa08a64d2bb7edcfbc85: Status 404 returned error can't find the container with id 9347604974e79d5ca3d674dcceb12464cf8ea43742a7aa08a64d2bb7edcfbc85 Nov 03 23:30:33.338911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:33.338879 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" event={"ID":"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d","Type":"ContainerStarted","Data":"9347604974e79d5ca3d674dcceb12464cf8ea43742a7aa08a64d2bb7edcfbc85"} Nov 03 23:30:34.346315 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:34.346238 5272 generic.go:356] "Generic (PLEG): container finished" podID="e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" containerID="2ba562d2a2db64f8829733c99d1434599c199ba8c16301472513abbb0644de77" exitCode=0 Nov 03 23:30:34.346650 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:34.346320 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" event={"ID":"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d","Type":"ContainerDied","Data":"2ba562d2a2db64f8829733c99d1434599c199ba8c16301472513abbb0644de77"} Nov 03 23:30:35.481600 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.481578 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:35.617350 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.617270 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-kube-api-access-j4l4g\" (UniqueName: \"kubernetes.io/projected/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-kube-api-access-j4l4g\") pod \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " Nov 03 23:30:35.617508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.617366 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-openshift-ca\") pod \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " Nov 03 23:30:35.617508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.617466 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-xks-ca\") pod \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\" (UID: \"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d\") " Nov 03 23:30:35.617700 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.617683 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-openshift-ca" pod "e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" (UID: "e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:35.617948 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.617928 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:35.619532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.619503 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-xks-ca" pod "e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" (UID: "e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:35.619639 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.619509 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-kube-api-access-j4l4g" pod "e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" (UID: "e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d"). InnerVolumeSpecName "kube-api-access-j4l4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:35.718515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.718483 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4l4g\" (UniqueName: \"kubernetes.io/projected/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-kube-api-access-j4l4g\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:35.718515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:35.718510 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:36.357915 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:36.357879 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" event={"ID":"e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d","Type":"ContainerDied","Data":"9347604974e79d5ca3d674dcceb12464cf8ea43742a7aa08a64d2bb7edcfbc85"} Nov 03 23:30:36.358122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:36.357921 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8" Nov 03 23:30:36.358122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:36.357930 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9347604974e79d5ca3d674dcceb12464cf8ea43742a7aa08a64d2bb7edcfbc85" Nov 03 23:30:44.142244 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:44.142211 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b"] Nov 03 23:30:44.825711 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:44.825676 5272 scope.go:122] "RemoveContainer" containerID="aa0d4cf890f50adf29da1fbfa9f5ebc6e554b231ec27ae11558e85d623c27938" Nov 03 23:30:44.834352 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:44.834331 5272 scope.go:122] "RemoveContainer" containerID="dbdf2b010af10ecbf6f0a51c002f0d3ba423b71ecb62cb06cea7faff2e4d811a" Nov 03 23:30:49.142662 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.142619 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" podUID="ad2ec4fd-5ba9-4e5d-b319-92c18699df53" containerName="payload-pre-processing" containerID="cri-o://b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c" gracePeriod=25 Nov 03 23:30:49.409651 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.409614 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:49.437870 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.437832 5272 generic.go:356] "Generic (PLEG): container finished" podID="ad2ec4fd-5ba9-4e5d-b319-92c18699df53" containerID="b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c" exitCode=0 Nov 03 23:30:49.438071 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.437929 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" event={"ID":"ad2ec4fd-5ba9-4e5d-b319-92c18699df53","Type":"ContainerDied","Data":"b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c"} Nov 03 23:30:49.438071 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.437982 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" event={"ID":"ad2ec4fd-5ba9-4e5d-b319-92c18699df53","Type":"ContainerDied","Data":"d0a6767c76f4a3ae1c70ce177a5671334f0f24e7373a528d2d97862a43042532"} Nov 03 23:30:49.438071 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.438011 5272 scope.go:122] "RemoveContainer" containerID="b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c" Nov 03 23:30:49.438265 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.438166 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b" Nov 03 23:30:49.456197 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.456171 5272 scope.go:122] "RemoveContainer" containerID="b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c" Nov 03 23:30:49.456707 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:30:49.456675 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c\": container with ID starting with b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c not found: ID does not exist" containerID="b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c" Nov 03 23:30:49.456934 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.456717 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c"} err="failed to get container status \"b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c\": rpc error: code = NotFound desc = could not find container \"b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c\": container with ID starting with b004192398ffedd6c4fc8275fc80b7584d6e12d927d82cb6e57d7a2794f7c76c not found: ID does not exist" Nov 03 23:30:49.543892 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.543856 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-kube-api-access-khs8z\" (UniqueName: \"kubernetes.io/projected/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-kube-api-access-khs8z\") pod \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " Nov 03 23:30:49.544096 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.543904 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-plugins-config-volume\") pod \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\" (UID: \"ad2ec4fd-5ba9-4e5d-b319-92c18699df53\") " Nov 03 23:30:49.544636 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.544573 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-plugins-config-volume" pod "ad2ec4fd-5ba9-4e5d-b319-92c18699df53" (UID: "ad2ec4fd-5ba9-4e5d-b319-92c18699df53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:49.546348 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.546319 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-kube-api-access-khs8z" pod "ad2ec4fd-5ba9-4e5d-b319-92c18699df53" (UID: "ad2ec4fd-5ba9-4e5d-b319-92c18699df53"). InnerVolumeSpecName "kube-api-access-khs8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:49.645795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.645727 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khs8z\" (UniqueName: \"kubernetes.io/projected/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-kube-api-access-khs8z\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:49.645795 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.645753 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2ec4fd-5ba9-4e5d-b319-92c18699df53-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:49.852767 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.852728 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b"] Nov 03 23:30:49.861215 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:49.861184 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc1c5661-585757c4f9-hqp2b"] Nov 03 23:30:51.192399 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:51.192362 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8"] Nov 03 23:30:51.238992 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:51.238961 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc1c5661-z8hn8"] Nov 03 23:30:51.491774 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:51.491690 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2ec4fd-5ba9-4e5d-b319-92c18699df53" path="/var/lib/kubelet/pods/ad2ec4fd-5ba9-4e5d-b319-92c18699df53/volumes" Nov 03 23:30:51.492211 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:51.492183 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" path="/var/lib/kubelet/pods/e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d/volumes" Nov 03 23:30:53.285337 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:53.285305 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz"] Nov 03 23:30:53.285813 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:53.285550 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" containerID="cri-o://8fd8fbe0d5278d95674ae5ae749c0bbf200f1d98fbac45372ce6081cf43771ec" gracePeriod=30 Nov 03 23:30:53.641300 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:53.641226 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:30:54.498992 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:30:54.498957 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:30:55.669279 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:55.669244 5272 patch_prober.go:29] interesting pod/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:30:55.669647 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:55.669309 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:30:58.483284 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.483253 5272 generic.go:356] "Generic (PLEG): container finished" podID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerID="8fd8fbe0d5278d95674ae5ae749c0bbf200f1d98fbac45372ce6081cf43771ec" exitCode=0 Nov 03 23:30:58.483666 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.483326 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" event={"ID":"665cc4fc-02b9-4455-880b-c4cb014c6f3d","Type":"ContainerDied","Data":"8fd8fbe0d5278d95674ae5ae749c0bbf200f1d98fbac45372ce6081cf43771ec"} Nov 03 23:30:58.483666 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.483386 5272 scope.go:122] "RemoveContainer" containerID="f5945af27ee107c18b506102c0a9dd94c8bf5055163954964e996c370a84802b" Nov 03 23:30:58.550327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.550274 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:30:58.636385 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636349 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-certs\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.636552 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636435 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-credential-socket\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.636552 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636506 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-data\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.636552 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636542 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-podinfo\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.636740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636583 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-envoy\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.636740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636632 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istiod-ca-cert\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.636740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636662 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-certs" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:58.636740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636673 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-socket\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.636740 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636721 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-credential-socket" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:58.637067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636759 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-kube-api-access-zqx85\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-kube-api-access-zqx85\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.637067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636866 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-token\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-token\") pod \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\" (UID: \"665cc4fc-02b9-4455-880b-c4cb014c6f3d\") " Nov 03 23:30:58.637067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636864 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-data" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:58.637067 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.636943 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-socket" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:58.637289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.637051 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istiod-ca-cert" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:58.637289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.637183 5272 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-data\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.637289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.637199 5272 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istiod-ca-cert\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.637289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.637212 5272 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.637289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.637232 5272 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-workload-certs\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.637289 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.637250 5272 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-credential-socket\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.638862 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.638837 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-kube-api-access-zqx85" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "kube-api-access-zqx85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:58.638985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.638869 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-podinfo" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:30:58.639042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.638985 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-token" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:58.639042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.639020 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-envoy" pod "665cc4fc-02b9-4455-880b-c4cb014c6f3d" (UID: "665cc4fc-02b9-4455-880b-c4cb014c6f3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:58.738612 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.738583 5272 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-podinfo\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.738612 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.738608 5272 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-envoy\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.738785 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.738619 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqx85\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-kube-api-access-zqx85\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:58.738785 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:58.738632 5272 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665cc4fc-02b9-4455-880b-c4cb014c6f3d-istio-token\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:59.488987 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:59.488959 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" Nov 03 23:30:59.491642 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:59.491613 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz" event={"ID":"665cc4fc-02b9-4455-880b-c4cb014c6f3d","Type":"ContainerDied","Data":"defea411350805303490e448d5665d0a88ff84bf83f32adf6a51277f0ac59087"} Nov 03 23:30:59.491714 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:59.491675 5272 scope.go:122] "RemoveContainer" containerID="8fd8fbe0d5278d95674ae5ae749c0bbf200f1d98fbac45372ce6081cf43771ec" Nov 03 23:30:59.689923 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:59.689889 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz"] Nov 03 23:30:59.697042 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:30:59.697006 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cc1c5661-openshift-default-65b64fd645-v2pmz"] Nov 03 23:31:01.490621 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:01.490590 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" path="/var/lib/kubelet/pods/665cc4fc-02b9-4455-880b-c4cb014c6f3d/volumes" Nov 03 23:31:03.040276 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040242 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m"] Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040633 5272 cpu_manager.go:414] "containerMap: removing container" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040645 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040658 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ad2ec4fd-5ba9-4e5d-b319-92c18699df53" containerName="payload-pre-processing" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040663 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2ec4fd-5ba9-4e5d-b319-92c18699df53" containerName="payload-pre-processing" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040674 5272 cpu_manager.go:414] "containerMap: removing container" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040679 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040692 5272 cpu_manager.go:414] "containerMap: removing container" podUID="e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" containerName="revoke-keys" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040697 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" containerName="revoke-keys" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040707 5272 cpu_manager.go:414] "containerMap: removing container" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:31:03.040726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040712 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:31:03.041119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040765 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8b1ca0-7e2e-42bd-bbd1-cb42e6043372" containerName="cleanup" Nov 03 23:31:03.041119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040775 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" Nov 03 23:31:03.041119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040782 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2ec4fd-5ba9-4e5d-b319-92c18699df53" containerName="payload-pre-processing" Nov 03 23:31:03.041119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040788 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="665cc4fc-02b9-4455-880b-c4cb014c6f3d" containerName="istio-proxy" Nov 03 23:31:03.041119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.040794 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a5c581-c755-4d1a-8f7f-7f1bcd227c8d" containerName="revoke-keys" Nov 03 23:31:03.045390 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.045369 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.097951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.097924 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m"] Nov 03 23:31:03.142891 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.142849 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-05b91b\"" Nov 03 23:31:03.143066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.143001 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-05b91b-dockercfg-vp6s5\"" Nov 03 23:31:03.182392 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.182349 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwx5\" (UniqueName: \"kubernetes.io/projected/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-kube-api-access-4kwx5\") pod \"payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.182596 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.182515 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.191912 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.191879 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p"] Nov 03 23:31:03.195939 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.195918 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.242224 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.242193 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p"] Nov 03 23:31:03.283550 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.283523 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwx5\" (UniqueName: \"kubernetes.io/projected/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-kube-api-access-4kwx5\") pod \"payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.283668 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.283560 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65390d40-4241-46fb-9953-3830f3f8bc1e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.283668 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.283594 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm54s\" (UniqueName: \"kubernetes.io/projected/65390d40-4241-46fb-9953-3830f3f8bc1e-kube-api-access-gm54s\") pod \"payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.283668 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.283644 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.284361 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.284335 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.340548 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.340486 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwx5\" (UniqueName: \"kubernetes.io/projected/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-kube-api-access-4kwx5\") pod \"payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.358282 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.358261 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:03.384658 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.384632 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65390d40-4241-46fb-9953-3830f3f8bc1e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.384762 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.384672 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm54s\" (UniqueName: \"kubernetes.io/projected/65390d40-4241-46fb-9953-3830f3f8bc1e-kube-api-access-gm54s\") pod \"payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.385272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.385252 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65390d40-4241-46fb-9953-3830f3f8bc1e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.442036 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.442009 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm54s\" (UniqueName: \"kubernetes.io/projected/65390d40-4241-46fb-9953-3830f3f8bc1e-kube-api-access-gm54s\") pod \"payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.510540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.510512 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:03.758303 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.758224 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m"] Nov 03 23:31:03.765060 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:31:03.765032 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65ee2ea_7d4b_48ad_9c07_7e25d9e7a607.slice/crio-4732306cb7bdcfc66671546ea2801c63cf4396e71889ebb84835f04f01154798 WatchSource:0}: Error finding container 4732306cb7bdcfc66671546ea2801c63cf4396e71889ebb84835f04f01154798: Status 404 returned error can't find the container with id 4732306cb7bdcfc66671546ea2801c63cf4396e71889ebb84835f04f01154798 Nov 03 23:31:03.894032 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:03.894000 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p"] Nov 03 23:31:03.901150 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:31:03.901122 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65390d40_4241_46fb_9953_3830f3f8bc1e.slice/crio-c73cdbecb258fe31f521a7fed5a83574ea727ddd53f8a184b4ffb9585d25f2d0 WatchSource:0}: Error finding container c73cdbecb258fe31f521a7fed5a83574ea727ddd53f8a184b4ffb9585d25f2d0: Status 404 returned error can't find the container with id c73cdbecb258fe31f521a7fed5a83574ea727ddd53f8a184b4ffb9585d25f2d0 Nov 03 23:31:04.520605 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.520568 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" event={"ID":"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607","Type":"ContainerStarted","Data":"aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03"} Nov 03 23:31:04.521143 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.520626 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" event={"ID":"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607","Type":"ContainerStarted","Data":"4732306cb7bdcfc66671546ea2801c63cf4396e71889ebb84835f04f01154798"} Nov 03 23:31:04.521143 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.520683 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:04.522219 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.522199 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" event={"ID":"65390d40-4241-46fb-9953-3830f3f8bc1e","Type":"ContainerStarted","Data":"1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9"} Nov 03 23:31:04.522334 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.522238 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" event={"ID":"65390d40-4241-46fb-9953-3830f3f8bc1e","Type":"ContainerStarted","Data":"c73cdbecb258fe31f521a7fed5a83574ea727ddd53f8a184b4ffb9585d25f2d0"} Nov 03 23:31:04.522334 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.522273 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:04.629368 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.629237 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" podStartSLOduration=1.629223418 podStartE2EDuration="1.629223418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:31:04.628748541 +0000 UTC m=+4045.251056852" watchObservedRunningTime="2026-11-03 23:31:04.629223418 +0000 UTC m=+4045.251531726" Nov 03 23:31:04.739789 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:04.738901 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" podStartSLOduration=1.738885567 podStartE2EDuration="1.738885567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:31:04.737154899 +0000 UTC m=+4045.359463207" watchObservedRunningTime="2026-11-03 23:31:04.738885567 +0000 UTC m=+4045.361193874" Nov 03 23:31:13.780629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:13.780591 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:31:15.529840 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:15.529791 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:31:15.530257 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:15.529887 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:31:20.045783 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:20.045704 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct"] Nov 03 23:31:20.092545 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:20.092517 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1a27b4-jbtct"] Nov 03 23:31:21.490962 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:31:21.490930 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dceb03-a50d-40cc-96d3-f939896bdf06" path="/var/lib/kubelet/pods/36dceb03-a50d-40cc-96d3-f939896bdf06/volumes" Nov 03 23:31:24.499402 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:31:24.499362 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:31:54.500282 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:31:54.500242 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:32:24.501023 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:32:24.500984 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:32:44.876515 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:44.876483 5272 scope.go:122] "RemoveContainer" containerID="afbb06aeaa4e35f399188124599e23a105e9407e4c08fe1fef5e84e4481d64ff" Nov 03 23:32:44.884759 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:44.884729 5272 scope.go:122] "RemoveContainer" containerID="659011caf716165ef8c876b7d5fa2dfe60d04d4c8d32716de9729643c39d18e1" Nov 03 23:32:54.499493 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:32:54.499411 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:32:57.577078 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.577045 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj"] Nov 03 23:32:57.580508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.580485 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.629601 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.629575 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:32:57.629736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.629720 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:32:57.676304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.676277 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:32:57.676304 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.676273 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fmpj4\"" Nov 03 23:32:57.678123 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.678101 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj"] Nov 03 23:32:57.754339 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.754306 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.754510 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.754458 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcw9p\" (UniqueName: \"kubernetes.io/projected/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-kube-api-access-xcw9p\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.754563 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.754518 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.855639 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.855548 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcw9p\" (UniqueName: \"kubernetes.io/projected/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-kube-api-access-xcw9p\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.855639 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.855601 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.855639 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.855636 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.856419 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.856389 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.857967 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.857945 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:57.912985 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:57.912951 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcw9p\" (UniqueName: \"kubernetes.io/projected/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-kube-api-access-xcw9p\") pod \"maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:58.192980 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:58.192885 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:32:58.566489 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:58.566452 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj"] Nov 03 23:32:59.068934 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:59.068887 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" event={"ID":"3d334cea-42b8-4d30-93ee-ad0cec5a90a6","Type":"ContainerStarted","Data":"50bf9bfe142f54f6ee83173ef8ef53c4263e46ba2ba3dd4c2981e55b9fea9fb5"} Nov 03 23:32:59.069347 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:59.068990 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" event={"ID":"3d334cea-42b8-4d30-93ee-ad0cec5a90a6","Type":"ContainerStarted","Data":"c26286fe2a6caaf7ea58a55184b42ad0e277e4315dc619027c29b353ac67e94d"} Nov 03 23:32:59.178688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:32:59.178585 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" podStartSLOduration=2.178573472 podStartE2EDuration="2.178573472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:32:59.176719708 +0000 UTC m=+4159.799028017" watchObservedRunningTime="2026-11-03 23:32:59.178573472 +0000 UTC m=+4159.800881774" Nov 03 23:33:00.075327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:00.075289 5272 generic.go:356] "Generic (PLEG): container finished" podID="3d334cea-42b8-4d30-93ee-ad0cec5a90a6" containerID="50bf9bfe142f54f6ee83173ef8ef53c4263e46ba2ba3dd4c2981e55b9fea9fb5" exitCode=0 Nov 03 23:33:00.075725 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:00.075351 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" event={"ID":"3d334cea-42b8-4d30-93ee-ad0cec5a90a6","Type":"ContainerDied","Data":"50bf9bfe142f54f6ee83173ef8ef53c4263e46ba2ba3dd4c2981e55b9fea9fb5"} Nov 03 23:33:01.216232 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.216207 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:33:01.390492 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.390420 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-openshift-ca\") pod \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " Nov 03 23:33:01.390492 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.390459 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-kube-api-access-xcw9p\" (UniqueName: \"kubernetes.io/projected/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-kube-api-access-xcw9p\") pod \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " Nov 03 23:33:01.390687 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.390499 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-xks-ca\") pod \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\" (UID: \"3d334cea-42b8-4d30-93ee-ad0cec5a90a6\") " Nov 03 23:33:01.390825 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.390786 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-openshift-ca" pod "3d334cea-42b8-4d30-93ee-ad0cec5a90a6" (UID: "3d334cea-42b8-4d30-93ee-ad0cec5a90a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:33:01.392593 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.392568 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-kube-api-access-xcw9p" pod "3d334cea-42b8-4d30-93ee-ad0cec5a90a6" (UID: "3d334cea-42b8-4d30-93ee-ad0cec5a90a6"). InnerVolumeSpecName "kube-api-access-xcw9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:01.392706 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.392601 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-xks-ca" pod "3d334cea-42b8-4d30-93ee-ad0cec5a90a6" (UID: "3d334cea-42b8-4d30-93ee-ad0cec5a90a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:33:01.491582 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.491546 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:01.491582 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.491576 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcw9p\" (UniqueName: \"kubernetes.io/projected/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-kube-api-access-xcw9p\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:01.491582 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:01.491587 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d334cea-42b8-4d30-93ee-ad0cec5a90a6-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:02.085891 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:02.085851 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" event={"ID":"3d334cea-42b8-4d30-93ee-ad0cec5a90a6","Type":"ContainerDied","Data":"c26286fe2a6caaf7ea58a55184b42ad0e277e4315dc619027c29b353ac67e94d"} Nov 03 23:33:02.086066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:02.085919 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26286fe2a6caaf7ea58a55184b42ad0e277e4315dc619027c29b353ac67e94d" Nov 03 23:33:02.086066 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:02.085873 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj" Nov 03 23:33:09.114089 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:09.114054 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p"] Nov 03 23:33:09.223589 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:09.223552 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m"] Nov 03 23:33:14.114734 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.114685 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" podUID="65390d40-4241-46fb-9953-3830f3f8bc1e" containerName="payload-processing" containerID="cri-o://1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9" gracePeriod=25 Nov 03 23:33:14.224726 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.224685 5272 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" podUID="b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" containerName="payload-pre-processing" containerID="cri-o://aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03" gracePeriod=25 Nov 03 23:33:14.377269 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.377202 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:33:14.430430 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.430396 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65390d40-4241-46fb-9953-3830f3f8bc1e-kube-api-access-gm54s\" (UniqueName: \"kubernetes.io/projected/65390d40-4241-46fb-9953-3830f3f8bc1e-kube-api-access-gm54s\") pod \"65390d40-4241-46fb-9953-3830f3f8bc1e\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " Nov 03 23:33:14.430625 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.430467 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65390d40-4241-46fb-9953-3830f3f8bc1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65390d40-4241-46fb-9953-3830f3f8bc1e-plugins-config-volume\") pod \"65390d40-4241-46fb-9953-3830f3f8bc1e\" (UID: \"65390d40-4241-46fb-9953-3830f3f8bc1e\") " Nov 03 23:33:14.430877 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.430850 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65390d40-4241-46fb-9953-3830f3f8bc1e-plugins-config-volume" pod "65390d40-4241-46fb-9953-3830f3f8bc1e" (UID: "65390d40-4241-46fb-9953-3830f3f8bc1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:33:14.432672 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.432650 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65390d40-4241-46fb-9953-3830f3f8bc1e-kube-api-access-gm54s" pod "65390d40-4241-46fb-9953-3830f3f8bc1e" (UID: "65390d40-4241-46fb-9953-3830f3f8bc1e"). InnerVolumeSpecName "kube-api-access-gm54s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:14.481568 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.481535 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:33:14.531921 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.531885 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-kube-api-access-4kwx5\" (UniqueName: \"kubernetes.io/projected/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-kube-api-access-4kwx5\") pod \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " Nov 03 23:33:14.532125 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.531932 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-plugins-config-volume\") pod \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\" (UID: \"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607\") " Nov 03 23:33:14.532312 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.532293 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65390d40-4241-46fb-9953-3830f3f8bc1e-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:14.532312 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.532315 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm54s\" (UniqueName: \"kubernetes.io/projected/65390d40-4241-46fb-9953-3830f3f8bc1e-kube-api-access-gm54s\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:14.532592 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.532439 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-plugins-config-volume" pod "b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" (UID: "b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:33:14.534131 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.534108 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-kube-api-access-4kwx5" pod "b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" (UID: "b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607"). InnerVolumeSpecName "kube-api-access-4kwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:14.633346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.633247 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kwx5\" (UniqueName: \"kubernetes.io/projected/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-kube-api-access-4kwx5\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:14.633346 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:14.633278 5272 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607-plugins-config-volume\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:15.148394 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.148350 5272 generic.go:356] "Generic (PLEG): container finished" podID="b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" containerID="aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03" exitCode=0 Nov 03 23:33:15.148900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.148408 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" Nov 03 23:33:15.148900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.148432 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" event={"ID":"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607","Type":"ContainerDied","Data":"aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03"} Nov 03 23:33:15.148900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.148491 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m" event={"ID":"b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607","Type":"ContainerDied","Data":"4732306cb7bdcfc66671546ea2801c63cf4396e71889ebb84835f04f01154798"} Nov 03 23:33:15.148900 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.148515 5272 scope.go:122] "RemoveContainer" containerID="aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03" Nov 03 23:33:15.149934 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.149885 5272 generic.go:356] "Generic (PLEG): container finished" podID="65390d40-4241-46fb-9953-3830f3f8bc1e" containerID="1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9" exitCode=0 Nov 03 23:33:15.150006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.149927 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" event={"ID":"65390d40-4241-46fb-9953-3830f3f8bc1e","Type":"ContainerDied","Data":"1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9"} Nov 03 23:33:15.150006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.149969 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" event={"ID":"65390d40-4241-46fb-9953-3830f3f8bc1e","Type":"ContainerDied","Data":"c73cdbecb258fe31f521a7fed5a83574ea727ddd53f8a184b4ffb9585d25f2d0"} Nov 03 23:33:15.150006 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.149934 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p" Nov 03 23:33:15.158182 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.158164 5272 scope.go:122] "RemoveContainer" containerID="aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03" Nov 03 23:33:15.158470 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:33:15.158453 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03\": container with ID starting with aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03 not found: ID does not exist" containerID="aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03" Nov 03 23:33:15.158542 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.158479 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03"} err="failed to get container status \"aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03\": rpc error: code = NotFound desc = could not find container \"aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03\": container with ID starting with aae55732e592d88a47f194ce51798a34ccfc628ca24ad2a6e182ece7e6030f03 not found: ID does not exist" Nov 03 23:33:15.158542 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.158495 5272 scope.go:122] "RemoveContainer" containerID="1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9" Nov 03 23:33:15.166614 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.166597 5272 scope.go:122] "RemoveContainer" containerID="1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9" Nov 03 23:33:15.166879 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:33:15.166861 5272 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9\": container with ID starting with 1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9 not found: ID does not exist" containerID="1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9" Nov 03 23:33:15.166933 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.166885 5272 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9"} err="failed to get container status \"1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9\": rpc error: code = NotFound desc = could not find container \"1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9\": container with ID starting with 1aa51821a0729966165e93beb753a85feb490141ac5ad030191dcb50a8aa9ca9 not found: ID does not exist" Nov 03 23:33:15.302165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.302127 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p"] Nov 03 23:33:15.348920 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.348890 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05b91b-66bb9db55d-gcw9p"] Nov 03 23:33:15.408372 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.408277 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m"] Nov 03 23:33:15.412710 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.412681 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05b91b-559cc4c5bc-pmj9m"] Nov 03 23:33:15.492124 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.492091 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65390d40-4241-46fb-9953-3830f3f8bc1e" path="/var/lib/kubelet/pods/65390d40-4241-46fb-9953-3830f3f8bc1e/volumes" Nov 03 23:33:15.492550 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:15.492532 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" path="/var/lib/kubelet/pods/b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607/volumes" Nov 03 23:33:18.239791 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:18.239755 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj"] Nov 03 23:33:18.286143 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:18.286110 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-05b91b-rxtqj"] Nov 03 23:33:19.491175 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:19.491141 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d334cea-42b8-4d30-93ee-ad0cec5a90a6" path="/var/lib/kubelet/pods/3d334cea-42b8-4d30-93ee-ad0cec5a90a6/volumes" Nov 03 23:33:23.040909 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:23.040876 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:33:24.508270 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:33:24.508228 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:33:28.044443 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:28.044403 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp"] Nov 03 23:33:28.090856 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:28.090794 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sqfnp"] Nov 03 23:33:29.491081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:29.491045 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8b7820-db4d-49fc-997d-299d93ea623e" path="/var/lib/kubelet/pods/ac8b7820-db4d-49fc-997d-299d93ea623e/volumes" Nov 03 23:33:36.153400 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:36.153359 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:33:48.450424 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:48.450393 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:33:54.499821 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:33:54.499771 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:33:58.315957 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.315922 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k"] Nov 03 23:33:58.316395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316301 5272 cpu_manager.go:414] "containerMap: removing container" podUID="65390d40-4241-46fb-9953-3830f3f8bc1e" containerName="payload-processing" Nov 03 23:33:58.316395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316313 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65390d40-4241-46fb-9953-3830f3f8bc1e" containerName="payload-processing" Nov 03 23:33:58.316395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316334 5272 cpu_manager.go:414] "containerMap: removing container" podUID="3d334cea-42b8-4d30-93ee-ad0cec5a90a6" containerName="revoke-keys" Nov 03 23:33:58.316395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316339 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d334cea-42b8-4d30-93ee-ad0cec5a90a6" containerName="revoke-keys" Nov 03 23:33:58.316395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316366 5272 cpu_manager.go:414] "containerMap: removing container" podUID="b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" containerName="payload-pre-processing" Nov 03 23:33:58.316395 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316372 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" containerName="payload-pre-processing" Nov 03 23:33:58.316639 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316429 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d334cea-42b8-4d30-93ee-ad0cec5a90a6" containerName="revoke-keys" Nov 03 23:33:58.316639 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316440 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65ee2ea-7d4b-48ad-9c07-7e25d9e7a607" containerName="payload-pre-processing" Nov 03 23:33:58.316639 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.316448 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="65390d40-4241-46fb-9953-3830f3f8bc1e" containerName="payload-processing" Nov 03 23:33:58.319558 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.319536 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.368593 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.368566 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:33:58.368736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.368681 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fmpj4\"" Nov 03 23:33:58.368736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.368688 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:33:58.368736 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.368684 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:33:58.374383 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.374355 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k"] Nov 03 23:33:58.496050 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.496017 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.496199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.496061 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.496199 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.496079 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfvv\" (UniqueName: \"kubernetes.io/projected/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-kube-api-access-2nfvv\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.597433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.597350 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.597433 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.597410 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.597679 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.597440 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfvv\" (UniqueName: \"kubernetes.io/projected/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-kube-api-access-2nfvv\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.598129 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.598102 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.599797 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.599777 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.655110 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.655084 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfvv\" (UniqueName: \"kubernetes.io/projected/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-kube-api-access-2nfvv\") pod \"maas-api-revoke-sub-f03629db-lgf4k\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:58.931864 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:58.931779 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:33:59.302575 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:59.302540 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k"] Nov 03 23:33:59.309841 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:33:59.309776 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467a74b3_dc5f_4ace_8eb0_850ebaedee0a.slice/crio-fb2434f22effdda688279e87ef774d7c95d00e1d34293300919fac85799db00c WatchSource:0}: Error finding container fb2434f22effdda688279e87ef774d7c95d00e1d34293300919fac85799db00c: Status 404 returned error can't find the container with id fb2434f22effdda688279e87ef774d7c95d00e1d34293300919fac85799db00c Nov 03 23:33:59.447334 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:33:59.447299 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" event={"ID":"467a74b3-dc5f-4ace-8eb0-850ebaedee0a","Type":"ContainerStarted","Data":"fb2434f22effdda688279e87ef774d7c95d00e1d34293300919fac85799db00c"} Nov 03 23:34:00.453979 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:00.453883 5272 generic.go:356] "Generic (PLEG): container finished" podID="467a74b3-dc5f-4ace-8eb0-850ebaedee0a" containerID="bd0bcb50c24ac17170f641d5db460579c5c6d55ae6b91bcbaa76e03daabbe9b3" exitCode=0 Nov 03 23:34:00.454438 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:00.453966 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" event={"ID":"467a74b3-dc5f-4ace-8eb0-850ebaedee0a","Type":"ContainerDied","Data":"bd0bcb50c24ac17170f641d5db460579c5c6d55ae6b91bcbaa76e03daabbe9b3"} Nov 03 23:34:01.589363 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.589338 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:34:01.622195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.622173 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-kube-api-access-2nfvv\" (UniqueName: \"kubernetes.io/projected/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-kube-api-access-2nfvv\") pod \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " Nov 03 23:34:01.622327 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.622279 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-xks-ca\") pod \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " Nov 03 23:34:01.622384 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.622357 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-openshift-ca\") pod \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\" (UID: \"467a74b3-dc5f-4ace-8eb0-850ebaedee0a\") " Nov 03 23:34:01.622755 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.622725 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-openshift-ca" pod "467a74b3-dc5f-4ace-8eb0-850ebaedee0a" (UID: "467a74b3-dc5f-4ace-8eb0-850ebaedee0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:34:01.624272 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.624242 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-xks-ca" pod "467a74b3-dc5f-4ace-8eb0-850ebaedee0a" (UID: "467a74b3-dc5f-4ace-8eb0-850ebaedee0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:34:01.624392 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.624368 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-kube-api-access-2nfvv" pod "467a74b3-dc5f-4ace-8eb0-850ebaedee0a" (UID: "467a74b3-dc5f-4ace-8eb0-850ebaedee0a"). InnerVolumeSpecName "kube-api-access-2nfvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:34:01.724203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.724112 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:01.724203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.724152 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfvv\" (UniqueName: \"kubernetes.io/projected/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-kube-api-access-2nfvv\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:01.724203 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:01.724168 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467a74b3-dc5f-4ace-8eb0-850ebaedee0a-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:02.464980 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:02.464947 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" Nov 03 23:34:02.465166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:02.464972 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgf4k" event={"ID":"467a74b3-dc5f-4ace-8eb0-850ebaedee0a","Type":"ContainerDied","Data":"fb2434f22effdda688279e87ef774d7c95d00e1d34293300919fac85799db00c"} Nov 03 23:34:02.465166 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:02.465029 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2434f22effdda688279e87ef774d7c95d00e1d34293300919fac85799db00c" Nov 03 23:34:03.044321 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:03.044287 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s"] Nov 03 23:34:03.090549 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:03.090518 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8c45s"] Nov 03 23:34:03.491370 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:03.491292 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d58f4c-4bc1-45d4-a458-ba2cd3f2feff" path="/var/lib/kubelet/pods/23d58f4c-4bc1-45d4-a458-ba2cd3f2feff/volumes" Nov 03 23:34:11.175697 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:11.175655 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:34:17.372829 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.372750 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t"] Nov 03 23:34:17.373405 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.373380 5272 cpu_manager.go:414] "containerMap: removing container" podUID="467a74b3-dc5f-4ace-8eb0-850ebaedee0a" containerName="revoke-keys" Nov 03 23:34:17.373535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.373405 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467a74b3-dc5f-4ace-8eb0-850ebaedee0a" containerName="revoke-keys" Nov 03 23:34:17.373535 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.373508 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="467a74b3-dc5f-4ace-8eb0-850ebaedee0a" containerName="revoke-keys" Nov 03 23:34:17.377105 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.377080 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.428086 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.428052 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:34:17.428285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.428082 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:34:17.466229 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.466189 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.466440 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.466275 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54hw\" (UniqueName: \"kubernetes.io/projected/322baf55-8ebc-41a3-ba44-795dc9cbde75-kube-api-access-l54hw\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.466440 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.466330 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.474827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.474616 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:34:17.474827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.474752 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fmpj4\"" Nov 03 23:34:17.477852 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.477791 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t"] Nov 03 23:34:17.567615 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.567562 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54hw\" (UniqueName: \"kubernetes.io/projected/322baf55-8ebc-41a3-ba44-795dc9cbde75-kube-api-access-l54hw\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.567827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.567635 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.567827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.567797 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.568919 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.568885 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.570966 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.570936 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.626140 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.626060 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54hw\" (UniqueName: \"kubernetes.io/projected/322baf55-8ebc-41a3-ba44-795dc9cbde75-kube-api-access-l54hw\") pod \"maas-api-revoke-sub-0165d7d2-fjc9t\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:17.698826 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:17.698007 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:18.346827 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:18.343296 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t"] Nov 03 23:34:18.485830 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:34:18.484510 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322baf55_8ebc_41a3_ba44_795dc9cbde75.slice/crio-2f7c8eed79da7fdd98177f1880fb517662eda1dabd7f91f3d471a483f3ed85b6 WatchSource:0}: Error finding container 2f7c8eed79da7fdd98177f1880fb517662eda1dabd7f91f3d471a483f3ed85b6: Status 404 returned error can't find the container with id 2f7c8eed79da7fdd98177f1880fb517662eda1dabd7f91f3d471a483f3ed85b6 Nov 03 23:34:18.595907 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:18.595761 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" event={"ID":"322baf55-8ebc-41a3-ba44-795dc9cbde75","Type":"ContainerStarted","Data":"2f7c8eed79da7fdd98177f1880fb517662eda1dabd7f91f3d471a483f3ed85b6"} Nov 03 23:34:19.604823 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:19.604753 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" event={"ID":"322baf55-8ebc-41a3-ba44-795dc9cbde75","Type":"ContainerStarted","Data":"549dc20fce9291bdfd801faa9c4fcf2bc78546ec6eef7f4a61d727b9d4f8086b"} Nov 03 23:34:20.610666 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:20.610624 5272 generic.go:356] "Generic (PLEG): container finished" podID="322baf55-8ebc-41a3-ba44-795dc9cbde75" containerID="549dc20fce9291bdfd801faa9c4fcf2bc78546ec6eef7f4a61d727b9d4f8086b" exitCode=0 Nov 03 23:34:20.611139 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:20.610713 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" event={"ID":"322baf55-8ebc-41a3-ba44-795dc9cbde75","Type":"ContainerDied","Data":"549dc20fce9291bdfd801faa9c4fcf2bc78546ec6eef7f4a61d727b9d4f8086b"} Nov 03 23:34:21.746122 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.746098 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:21.771169 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.771143 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-openshift-ca\") pod \"322baf55-8ebc-41a3-ba44-795dc9cbde75\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " Nov 03 23:34:21.771266 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.771181 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322baf55-8ebc-41a3-ba44-795dc9cbde75-kube-api-access-l54hw\" (UniqueName: \"kubernetes.io/projected/322baf55-8ebc-41a3-ba44-795dc9cbde75-kube-api-access-l54hw\") pod \"322baf55-8ebc-41a3-ba44-795dc9cbde75\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " Nov 03 23:34:21.771266 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.771236 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-xks-ca\") pod \"322baf55-8ebc-41a3-ba44-795dc9cbde75\" (UID: \"322baf55-8ebc-41a3-ba44-795dc9cbde75\") " Nov 03 23:34:21.771517 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.771496 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-openshift-ca" pod "322baf55-8ebc-41a3-ba44-795dc9cbde75" (UID: "322baf55-8ebc-41a3-ba44-795dc9cbde75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:34:21.771629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.771608 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:21.773170 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.773149 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322baf55-8ebc-41a3-ba44-795dc9cbde75-kube-api-access-l54hw" pod "322baf55-8ebc-41a3-ba44-795dc9cbde75" (UID: "322baf55-8ebc-41a3-ba44-795dc9cbde75"). InnerVolumeSpecName "kube-api-access-l54hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:34:21.773295 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.773280 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-xks-ca" pod "322baf55-8ebc-41a3-ba44-795dc9cbde75" (UID: "322baf55-8ebc-41a3-ba44-795dc9cbde75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:34:21.872156 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.872091 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l54hw\" (UniqueName: \"kubernetes.io/projected/322baf55-8ebc-41a3-ba44-795dc9cbde75-kube-api-access-l54hw\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:21.872156 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:21.872118 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322baf55-8ebc-41a3-ba44-795dc9cbde75-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:22.621629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:22.621600 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" Nov 03 23:34:22.621629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:22.621611 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjc9t" event={"ID":"322baf55-8ebc-41a3-ba44-795dc9cbde75","Type":"ContainerDied","Data":"2f7c8eed79da7fdd98177f1880fb517662eda1dabd7f91f3d471a483f3ed85b6"} Nov 03 23:34:22.621873 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:22.621659 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7c8eed79da7fdd98177f1880fb517662eda1dabd7f91f3d471a483f3ed85b6" Nov 03 23:34:24.499075 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:34:24.499045 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:34:29.078833 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.078784 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh"] Nov 03 23:34:29.079472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.079200 5272 cpu_manager.go:414] "containerMap: removing container" podUID="322baf55-8ebc-41a3-ba44-795dc9cbde75" containerName="revoke-keys" Nov 03 23:34:29.079472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.079212 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322baf55-8ebc-41a3-ba44-795dc9cbde75" containerName="revoke-keys" Nov 03 23:34:29.079472 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.079278 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="322baf55-8ebc-41a3-ba44-795dc9cbde75" containerName="revoke-keys" Nov 03 23:34:29.082262 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.082241 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.131206 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.131176 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:34:29.131345 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.131265 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:34:29.134787 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.134754 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.134911 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.134886 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wnwb\" (UniqueName: \"kubernetes.io/projected/488cf9ba-24ae-49be-9230-a57c714e105f-kube-api-access-6wnwb\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.135077 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.134963 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.177838 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.177792 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fmpj4\"" Nov 03 23:34:29.177990 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.177869 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:34:29.179404 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.179381 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh"] Nov 03 23:34:29.235522 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.235504 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.235652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.235534 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wnwb\" (UniqueName: \"kubernetes.io/projected/488cf9ba-24ae-49be-9230-a57c714e105f-kube-api-access-6wnwb\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.235652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.235554 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.236274 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.236252 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.237951 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.237929 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.293221 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.293200 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wnwb\" (UniqueName: \"kubernetes.io/projected/488cf9ba-24ae-49be-9230-a57c714e105f-kube-api-access-6wnwb\") pod \"maas-api-revoke-sub-6b19338f-p6xhh\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.395712 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.395642 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:29.764450 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:29.764367 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh"] Nov 03 23:34:29.771731 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:34:29.771701 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488cf9ba_24ae_49be_9230_a57c714e105f.slice/crio-cadfa25b6d820183bbe758db040d7095bffdf6aec574f215c6217bcbac7658d0 WatchSource:0}: Error finding container cadfa25b6d820183bbe758db040d7095bffdf6aec574f215c6217bcbac7658d0: Status 404 returned error can't find the container with id cadfa25b6d820183bbe758db040d7095bffdf6aec574f215c6217bcbac7658d0 Nov 03 23:34:30.663062 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:30.663030 5272 generic.go:356] "Generic (PLEG): container finished" podID="488cf9ba-24ae-49be-9230-a57c714e105f" containerID="99dab0db8bd91812ce0306bb15c806d0324ade143d9a7e067445099900d22136" exitCode=0 Nov 03 23:34:30.663373 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:30.663107 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" event={"ID":"488cf9ba-24ae-49be-9230-a57c714e105f","Type":"ContainerDied","Data":"99dab0db8bd91812ce0306bb15c806d0324ade143d9a7e067445099900d22136"} Nov 03 23:34:30.663373 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:30.663161 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" event={"ID":"488cf9ba-24ae-49be-9230-a57c714e105f","Type":"ContainerStarted","Data":"cadfa25b6d820183bbe758db040d7095bffdf6aec574f215c6217bcbac7658d0"} Nov 03 23:34:31.798796 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.798772 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:31.862586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.862561 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-xks-ca\") pod \"488cf9ba-24ae-49be-9230-a57c714e105f\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " Nov 03 23:34:31.862713 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.862612 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488cf9ba-24ae-49be-9230-a57c714e105f-kube-api-access-6wnwb\" (UniqueName: \"kubernetes.io/projected/488cf9ba-24ae-49be-9230-a57c714e105f-kube-api-access-6wnwb\") pod \"488cf9ba-24ae-49be-9230-a57c714e105f\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " Nov 03 23:34:31.862713 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.862659 5272 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-openshift-ca\") pod \"488cf9ba-24ae-49be-9230-a57c714e105f\" (UID: \"488cf9ba-24ae-49be-9230-a57c714e105f\") " Nov 03 23:34:31.862993 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.862963 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-openshift-ca" pod "488cf9ba-24ae-49be-9230-a57c714e105f" (UID: "488cf9ba-24ae-49be-9230-a57c714e105f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:34:31.864681 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.864652 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-xks-ca" pod "488cf9ba-24ae-49be-9230-a57c714e105f" (UID: "488cf9ba-24ae-49be-9230-a57c714e105f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:34:31.864777 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.864729 5272 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488cf9ba-24ae-49be-9230-a57c714e105f-kube-api-access-6wnwb" pod "488cf9ba-24ae-49be-9230-a57c714e105f" (UID: "488cf9ba-24ae-49be-9230-a57c714e105f"). InnerVolumeSpecName "kube-api-access-6wnwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:34:31.964160 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.964103 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-xks-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:31.964160 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.964129 5272 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wnwb\" (UniqueName: \"kubernetes.io/projected/488cf9ba-24ae-49be-9230-a57c714e105f-kube-api-access-6wnwb\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:31.964160 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:31.964140 5272 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/488cf9ba-24ae-49be-9230-a57c714e105f-maas-api-openshift-ca\") on node \"ip-10-0-128-222.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:34:32.674342 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:32.674308 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" event={"ID":"488cf9ba-24ae-49be-9230-a57c714e105f","Type":"ContainerDied","Data":"cadfa25b6d820183bbe758db040d7095bffdf6aec574f215c6217bcbac7658d0"} Nov 03 23:34:32.674342 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:32.674338 5272 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p6xhh" Nov 03 23:34:32.674633 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:32.674359 5272 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cadfa25b6d820183bbe758db040d7095bffdf6aec574f215c6217bcbac7658d0" Nov 03 23:34:36.047576 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:36.047538 5272 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh"] Nov 03 23:34:36.093695 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:36.093664 5272 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da690c1c-khvkh"] Nov 03 23:34:37.491501 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:37.491465 5272 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521f94a0-4fac-436e-9ca4-20faa32666bc" path="/var/lib/kubelet/pods/521f94a0-4fac-436e-9ca4-20faa32666bc/volumes" Nov 03 23:34:41.437533 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:41.437502 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77d677f8bf-cvr95_86d55195-a812-4525-bc69-1419a63288b9/authorino/0.log" Nov 03 23:34:41.857428 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:41.857393 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2zdxm"] Nov 03 23:34:44.471836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:44.470894 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-79h22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:34:44.471836 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:44.470971 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-79h22" podUID="cbd938fa-3e39-4e39-a30e-4d2fa083d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:34:44.549828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:44.549506 5272 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qrmft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:34:44.549828 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:44.549665 5272 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qrmft" podUID="c047281a-ef57-4b5b-9db3-4947c5dbe7aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:34:44.921603 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:34:44.921573 5272 scope.go:122] "RemoveContainer" containerID="81885b001c77c704d01df1a5a5b441f95472c790f63b6c89db0481a6945218ce" Nov 03 23:34:54.499189 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:34:54.499150 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:35:02.148119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:02.148084 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77fffc8496-hf9sf_716b498c-8fc0-4470-a731-689814b925d5/manager/0.log" Nov 03 23:35:03.990559 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:03.990525 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-82qwd_14e070a9-8f8f-413b-9e84-bb8c44d4feca/server/0.log" Nov 03 23:35:05.909637 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:05.909604 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fjc9t_322baf55-8ebc-41a3-ba44-795dc9cbde75/revoke-keys/0.log" Nov 03 23:35:06.150050 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:06.150017 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-p6xhh_488cf9ba-24ae-49be-9230-a57c714e105f/revoke-keys/0.log" Nov 03 23:35:06.380954 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:06.380925 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lgf4k_467a74b3-dc5f-4ace-8eb0-850ebaedee0a/revoke-keys/0.log" Nov 03 23:35:08.178586 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:08.178553 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77d677f8bf-cvr95_86d55195-a812-4525-bc69-1419a63288b9/authorino/0.log" Nov 03 23:35:09.119190 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:09.119159 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8p92b_9c84240c-84ab-43b3-ba71-1235367c0987/kuadrant-console-plugin/0.log" Nov 03 23:35:09.892397 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:09.892368 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2zdxm_c975eec3-d424-46fc-9518-172b607bd071/limitador/0.log" Nov 03 23:35:11.469357 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:11.469324 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8kwp6_ed95a603-37ae-43d0-a834-8598e3cd3ceb/discovery/0.log" Nov 03 23:35:12.009540 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:12.009510 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84cff947f5-6bglq_204b7f88-d17a-4326-a5cf-9ef201d58871/kube-auth-proxy/0.log" Nov 03 23:35:12.468056 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:12.468023 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-79h22_cbd938fa-3e39-4e39-a30e-4d2fa083d588/istio-proxy/0.log" Nov 03 23:35:14.107688 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:14.107657 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-qrmft_c047281a-ef57-4b5b-9db3-4947c5dbe7aa/istio-proxy/0.log" Nov 03 23:35:14.347974 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:14.347940 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-rlkdl_ed051394-e221-4f2a-bf68-988385bea856/istio-proxy/0.log" Nov 03 23:35:14.581414 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:14.581387 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-wkwkn_1461ab8f-e9cf-409b-ad59-2bc90b14463f/istio-proxy/0.log" Nov 03 23:35:24.499787 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:35:24.499749 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:35:25.033902 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:25.033871 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-wlsbd_784bc8ce-c36d-4bcf-adc5-c58564276bea/payload-pre-processing/0.log" Nov 03 23:35:25.271633 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:25.271594 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6xcbp_a43c630f-c23e-4a54-bf40-4b0e9aaf17a7/payload-processing/0.log" Nov 03 23:35:26.467167 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:26.467138 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn_1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d/main/0.log" Nov 03 23:35:27.022578 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:27.022545 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-m59rp_ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0/main/0.log" Nov 03 23:35:27.491658 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:27.491623 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw_ca3f872d-bbbb-43e8-bed7-5e6c065a625f/main/0.log" Nov 03 23:35:27.736084 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:27.736048 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ggsmt_7f7a50d4-86b8-404f-99db-2b3be5d3c7cf/main/0.log" Nov 03 23:35:29.227069 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:29.227035 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77d677f8bf-cvr95_86d55195-a812-4525-bc69-1419a63288b9/authorino/0.log" Nov 03 23:35:44.941422 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:44.941385 5272 scope.go:122] "RemoveContainer" containerID="fe81cea999602c60470851a29ef84e5c562f6aee5d4610c027369feccce1f615" Nov 03 23:35:44.949739 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:35:44.949711 5272 scope.go:122] "RemoveContainer" containerID="3c36e52cff759cd0c4e6648eb8c1477cb799842c8f997cf77a5d57029e63e49a" Nov 03 23:35:54.499586 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:35:54.499507 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:36:18.534108 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:18.534071 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77fffc8496-hf9sf_716b498c-8fc0-4470-a731-689814b925d5/manager/0.log" Nov 03 23:36:20.009629 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:20.009601 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-82qwd_14e070a9-8f8f-413b-9e84-bb8c44d4feca/server/0.log" Nov 03 23:36:21.808119 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:21.808089 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fjc9t_322baf55-8ebc-41a3-ba44-795dc9cbde75/revoke-keys/0.log" Nov 03 23:36:22.031750 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:22.031720 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-p6xhh_488cf9ba-24ae-49be-9230-a57c714e105f/revoke-keys/0.log" Nov 03 23:36:22.259500 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:22.259424 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lgf4k_467a74b3-dc5f-4ace-8eb0-850ebaedee0a/revoke-keys/0.log" Nov 03 23:36:23.708073 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:23.708040 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77d677f8bf-cvr95_86d55195-a812-4525-bc69-1419a63288b9/authorino/0.log" Nov 03 23:36:24.499710 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:36:24.499673 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:36:24.657165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:24.657135 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8p92b_9c84240c-84ab-43b3-ba71-1235367c0987/kuadrant-console-plugin/0.log" Nov 03 23:36:25.348878 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:25.348842 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2zdxm_c975eec3-d424-46fc-9518-172b607bd071/limitador/0.log" Nov 03 23:36:26.493820 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:26.493778 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8kwp6_ed95a603-37ae-43d0-a834-8598e3cd3ceb/discovery/0.log" Nov 03 23:36:26.993342 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:26.993317 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84cff947f5-6bglq_204b7f88-d17a-4326-a5cf-9ef201d58871/kube-auth-proxy/0.log" Nov 03 23:36:27.429048 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:27.429014 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-79h22_cbd938fa-3e39-4e39-a30e-4d2fa083d588/istio-proxy/0.log" Nov 03 23:36:28.940261 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:28.940221 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-qrmft_c047281a-ef57-4b5b-9db3-4947c5dbe7aa/istio-proxy/0.log" Nov 03 23:36:29.160237 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:29.160203 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-rlkdl_ed051394-e221-4f2a-bf68-988385bea856/istio-proxy/0.log" Nov 03 23:36:29.387963 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:29.387922 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-wkwkn_1461ab8f-e9cf-409b-ad59-2bc90b14463f/istio-proxy/0.log" Nov 03 23:36:29.830686 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:29.830656 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-wlsbd_784bc8ce-c36d-4bcf-adc5-c58564276bea/payload-pre-processing/0.log" Nov 03 23:36:30.060386 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:30.060356 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6xcbp_a43c630f-c23e-4a54-bf40-4b0e9aaf17a7/payload-processing/0.log" Nov 03 23:36:30.950386 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:30.950354 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-v88hn_1e27d43a-4ac0-4f7d-87ab-3f74acbbda2d/main/0.log" Nov 03 23:36:31.513419 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:31.513390 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-m59rp_ba2a7cfe-c63c-4ea0-9b6a-8dc5974392a0/main/0.log" Nov 03 23:36:31.973996 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:31.973963 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gl9cw_ca3f872d-bbbb-43e8-bed7-5e6c065a625f/main/0.log" Nov 03 23:36:32.209861 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:32.209832 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ggsmt_7f7a50d4-86b8-404f-99db-2b3be5d3c7cf/main/0.log" Nov 03 23:36:44.975743 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:36:44.975709 5272 scope.go:122] "RemoveContainer" containerID="2ba562d2a2db64f8829733c99d1434599c199ba8c16301472513abbb0644de77" Nov 03 23:36:54.500001 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:36:54.499963 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:37:11.310841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:37:11.310793 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6xcbp_a43c630f-c23e-4a54-bf40-4b0e9aaf17a7/payload-processing/0.log" Nov 03 23:37:11.540081 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:37:11.540055 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6xcbp_a43c630f-c23e-4a54-bf40-4b0e9aaf17a7/payload-processing/0.log" Nov 03 23:37:24.499453 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:37:24.499371 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:37:54.499479 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:37:54.499445 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:38:18.055374 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:18.055343 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6xcbp_a43c630f-c23e-4a54-bf40-4b0e9aaf17a7/payload-processing/0.log" Nov 03 23:38:18.266782 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:18.266753 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6xcbp_a43c630f-c23e-4a54-bf40-4b0e9aaf17a7/payload-processing/0.log" Nov 03 23:38:24.499649 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:38:24.499615 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor" Nov 03 23:38:28.487635 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:28.487604 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zlkqx_b2e63ee2-e60b-45fd-90e4-e60785022ff7/global-pull-secret-syncer/0.log" Nov 03 23:38:30.810285 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:30.810251 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lrdk5_de70e8cd-d2c4-45b4-bcd4-9a726497f781/konnectivity-agent/0.log" Nov 03 23:38:31.728652 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:31.728616 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-222.us-west-2.compute.internal_d79e0f53b5b82fd1a135f4636735856a/haproxy/0.log" Nov 03 23:38:37.534612 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.534577 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77d677f8bf-cvr95_86d55195-a812-4525-bc69-1419a63288b9/authorino/0.log" Nov 03 23:38:37.896398 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.896299 5272 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-md998/perf-node-gather-daemonset-2cr97"] Nov 03 23:38:37.896729 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.896715 5272 cpu_manager.go:414] "containerMap: removing container" podUID="488cf9ba-24ae-49be-9230-a57c714e105f" containerName="revoke-keys" Nov 03 23:38:37.896774 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.896730 5272 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488cf9ba-24ae-49be-9230-a57c714e105f" containerName="revoke-keys" Nov 03 23:38:37.896841 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.896830 5272 memory_manager.go:365] "RemoveStaleState removing state" podUID="488cf9ba-24ae-49be-9230-a57c714e105f" containerName="revoke-keys" Nov 03 23:38:37.900013 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.899989 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:37.949256 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.949220 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-md998\"/\"kube-root-ca.crt\"" Nov 03 23:38:37.995654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.995619 5272 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-md998\"/\"openshift-service-ca.crt\"" Nov 03 23:38:37.995654 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.995630 5272 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-md998\"/\"default-dockercfg-sg8mv\"" Nov 03 23:38:37.997298 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:37.997273 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-md998/perf-node-gather-daemonset-2cr97"] Nov 03 23:38:38.043195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.043160 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-proc\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.043195 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.043199 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-lib-modules\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.043399 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.043360 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhd6\" (UniqueName: \"kubernetes.io/projected/a7305bdc-d860-4609-b8e3-e70462b247bf-kube-api-access-wwhd6\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.043399 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.043386 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-podres\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.043476 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.043401 5272 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-sys\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.143940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.143905 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhd6\" (UniqueName: \"kubernetes.io/projected/a7305bdc-d860-4609-b8e3-e70462b247bf-kube-api-access-wwhd6\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.143940 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.143946 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-podres\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.144207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.143963 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-sys\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.144207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.144007 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-proc\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.144207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.144032 5272 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-lib-modules\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.144207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.144063 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-podres\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.144207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.144130 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-sys\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.144207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.144151 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-lib-modules\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.144207 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.144137 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7305bdc-d860-4609-b8e3-e70462b247bf-proc\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.201611 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.201538 5272 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhd6\" (UniqueName: \"kubernetes.io/projected/a7305bdc-d860-4609-b8e3-e70462b247bf-kube-api-access-wwhd6\") pod \"perf-node-gather-daemonset-2cr97\" (UID: \"a7305bdc-d860-4609-b8e3-e70462b247bf\") " pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.212565 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.212539 5272 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.585051 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.585013 5272 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-md998/perf-node-gather-daemonset-2cr97"] Nov 03 23:38:38.592340 ip-10-0-128-222 kubenswrapper[5272]: W1103 23:38:38.592312 5272 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda7305bdc_d860_4609_b8e3_e70462b247bf.slice/crio-37ffaad417e14dfca1cba9bb0378e8919f5c526d0c1d8029db5e4f9759375fcf WatchSource:0}: Error finding container 37ffaad417e14dfca1cba9bb0378e8919f5c526d0c1d8029db5e4f9759375fcf: Status 404 returned error can't find the container with id 37ffaad417e14dfca1cba9bb0378e8919f5c526d0c1d8029db5e4f9759375fcf Nov 03 23:38:38.806670 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.806641 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8p92b_9c84240c-84ab-43b3-ba71-1235367c0987/kuadrant-console-plugin/0.log" Nov 03 23:38:38.873470 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.873378 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" event={"ID":"a7305bdc-d860-4609-b8e3-e70462b247bf","Type":"ContainerStarted","Data":"888239fa470ba3a5802668da1bb5a5a51f038e81d0624b6f9ac1b7152aef980b"} Nov 03 23:38:38.873470 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.873439 5272 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" event={"ID":"a7305bdc-d860-4609-b8e3-e70462b247bf","Type":"ContainerStarted","Data":"37ffaad417e14dfca1cba9bb0378e8919f5c526d0c1d8029db5e4f9759375fcf"} Nov 03 23:38:38.873666 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.873589 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:38.982532 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:38.982471 5272 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" podStartSLOduration=1.982454132 podStartE2EDuration="1.982454132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:38.980724484 +0000 UTC m=+4499.603032803" watchObservedRunningTime="2026-11-03 23:38:38.982454132 +0000 UTC m=+4499.604762440" Nov 03 23:38:39.684446 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:39.684407 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2zdxm_c975eec3-d424-46fc-9518-172b607bd071/limitador/0.log" Nov 03 23:38:42.720351 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:42.720324 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049bd4-c939-46bd-9162-5f25c25bc97d/alertmanager/0.log" Nov 03 23:38:43.139681 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:43.139651 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049bd4-c939-46bd-9162-5f25c25bc97d/config-reloader/0.log" Nov 03 23:38:43.560301 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:43.560270 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049bd4-c939-46bd-9162-5f25c25bc97d/kube-rbac-proxy-web/0.log" Nov 03 23:38:43.980133 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:43.980058 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049bd4-c939-46bd-9162-5f25c25bc97d/kube-rbac-proxy/0.log" Nov 03 23:38:44.399747 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:44.399715 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049bd4-c939-46bd-9162-5f25c25bc97d/kube-rbac-proxy-metric/0.log" Nov 03 23:38:44.820507 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:44.820476 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049bd4-c939-46bd-9162-5f25c25bc97d/prom-label-proxy/0.log" Nov 03 23:38:44.889582 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:44.889555 5272 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-md998/perf-node-gather-daemonset-2cr97" Nov 03 23:38:45.240941 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:45.240868 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049bd4-c939-46bd-9162-5f25c25bc97d/init-config-reloader/0.log" Nov 03 23:38:46.173461 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:46.173425 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-m5xmb_2ac641f1-837b-47f2-8aaf-09d93df2be1b/kube-state-metrics/0.log" Nov 03 23:38:46.591987 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:46.591956 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-m5xmb_2ac641f1-837b-47f2-8aaf-09d93df2be1b/kube-rbac-proxy-main/0.log" Nov 03 23:38:47.036165 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:47.036123 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-m5xmb_2ac641f1-837b-47f2-8aaf-09d93df2be1b/kube-rbac-proxy-self/0.log" Nov 03 23:38:48.347779 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:48.347750 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcrp2_838971d1-4352-4fa8-80ec-f45ceb86b53c/node-exporter/0.log" Nov 03 23:38:48.771617 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:48.771537 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcrp2_838971d1-4352-4fa8-80ec-f45ceb86b53c/kube-rbac-proxy/0.log" Nov 03 23:38:49.191508 ip-10-0-128-222 kubenswrapper[5272]: I1103 23:38:49.191471 5272 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gcrp2_838971d1-4352-4fa8-80ec-f45ceb86b53c/init-textfile/0.log" Nov 03 23:38:54.499895 ip-10-0-128-222 kubenswrapper[5272]: E1103 23:38:54.499797 5272 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-54cfh" containerName="extractor"